StreamGoose/README.md at 024ec03ba90958e485f1d1e1fa509373ed4fc497

Files

T

Luxferre 024ec03ba9 Updated the readme to reflect on the SSE change; cross-compilation also works now

2024-09-14 20:51:18 +03:00

17 KiB

Raw Blame History

StreamGoose (aka StreamGoOSE, Streamer's Go-based Open Source Enhancer) is an open tool to solve the problem of aggregating several live chats to display on a livestream.

As of now, most of the StreamGoose codebase is written in Go 1.23.1. The Web part is written in HTML5, CSS3 and JavaScript using the ES2015 standard.

Current features

The status of StreamGoose is currently alpha. Use at your own risk.

Nevertheless, the following features are already implemented:

Twitch and YouTube chat aggregation (see "Backend implementation status" section)
Handling Twitch emotes
Handling YouTube non-Unicode emotes (see "Backend implementation status" section)
Message styling based on Twitch and YouTube roles
Twitch username styling based on color preference
Message highlighting based on regular expressions
Third-party action (/me) command handling
YouTube avatar icon support
Twitch system message support (needs more testing)
Noto Color Emoji font bundled for universal Unicode emoji support

Planned features

Moving to OAuth 2.0 for YouTube
"Deleted message" support for YouTube
Automated Twitch chat badges output (global and channel-specific)
Internal chat ranking system by username
Backend-independent support for oldschool plaintext emoticons ;-)
Starting window geometry customization via config (now doable via custom JS)
...

Architecture

StreamGoose consists of three components: a set of backends, a message broker server and a Web-based frontend.

Message broker server

This is the main StreamGoose component. It provides three main functions:

accepting new messages from backends via external (POST /new) and internal (message_transport Go channel) ways;
providing the unified message queue for the frontend side consumption;
serving static files that make up the frontend (index.html, script.js, style.css and images) over the plain HTTP protocol.

Additionally, the server binary also launches the backends it knows about and opens a simple GUI window to show the Web content locally, feeding its unified output into the JS side via server-sent events. The latter part became possible thanks to the webview_go library.

Backends

StreamGoose can operate with two kinds of backends: internal and external.

An internal StreamGoose backend is a Go program embedded as a goroutine into the main binary. It communicates with the message broker by sending messages to the message_transport Go channel. The messages are defined using this Go structure:

type ChatMessage struct {
  Id uint64        `json:"id"`        /* message id (auto-filled) */
  Type string      `json:"type"`      /* message type string */
  Timestamp string `json:"timestamp"` /* timestamp string (formatted) */
  Username string  `json:"username"`  /* username */
  Text string      `json:"text"`      /* message text */
  Icon string      `json:"icon"`      /* optional message icon URI */
}

The Id field is populated automatically whenever a message enters the broker. For others, see below for the detailed description of the corresponding JSON fields in the external backends.

An external StreamGoose backend is a program (written in any network-enabled programming language) that can make a POST request to the corresponding message broker URL (http://[broker_host]:60053/new endpoint). The request body shall be a JSON object with the following fields (all passed as strings):

type: message type, consisting of hyphenated classes (e.g. yt-moderator or tw-vip-color#A1B2C3 etc, see below in the "Customization" section of this README). This field defines how the message will be styled when being rendered on the frontend.
timestamp: an ISO 8601 UTC timestamp string denoting when the message was sent. The only recommended format (in strftime syntax) is %Y-%m-%dT%H:%M:%SZ.
username: the message author name that should be displayed in the chat.
text: the message text that should be displayed in the chat.
icon: an optional icon URL passed to the chat if the message author has a personal user profile picture (avatar). This field can be left empty but should be included in the request anyway.

Whenever a new JSON message is shaped with the fields above, it shall be sent to the broker at the /new endpoint with the HTTP POST method. A {"status":1} response will be sent back if the request is successful.

As of now, StreamGoose provides two reference backends: for Twitch and for YouTube, both internal and written in Go. See "Backend implementation status" section for current notes and caveats.

Frontend

The StreamGoose frontend is a Web-based (HTML5) application served by the message broker server. It is opened in the built-in WebView-based GUI and provides a constantly updated aggregated live chat display, message type dependent styling, emote handling, custom highlighting and so on. The users are free and even encouraged to modify the web assets located in the /webroot directory to fit the chat to their own visual preferences.

Building and installation

Linux

You need to have Go 1.23.1 and above, a POSIX-compatible "make" as well as the webkit2gtk-4.1 dev package.

Just run make and get all the resulting files in the out directory.

macOS

TBD

Windows x64 (native)

You need to have Go 1.23.1 and above and a POSIX-compatible "make" in your %PATH%.

Just run make win64 and get all the resulting files in the out directory.

Windows x64 (cross-compiling from Linux)

Install the mingw-w64 compiler package.
Run make win64-cross

If the compiler complains about the EventToken.h file when building webview_go dependency, locate the include files and symlink it to the real eventtoken.h like this (as root):

# ln -s /usr/x86_64-w64-mingw32/include/eventtoken.h /usr/x86_64-w64-mingw32/include/EventToken.h

Then rerun the command from step 2.

Note: When testing the cross-compiled build under Wine, you must have Microsoft Edge WebView2 Runtime installed in order for the GUI to work at all. You can download this engine directly from the Microsoft website (select "Evergreen Standalone Installer") and install it into your Wine profile. Be sure to enable at least Windows 7 emulation in winecfg.

Usage

Enter the distribution directory and run the ./streamgoose or streamgoose.exe binary. It should run with the default config.json file, start the server, spawn the internal backends and open the UI window. Alternatively, you can pass another JSON file as the first command line parameter.

Configuration

StreamGoose is mostly configured through the config.json file distributed in the same directory as the main binary.

The following fields in the config.json are backend-agnostic:

server_port: a local port to run the HTTP REST server on. Default value: 60053 (for "GOOSE").
server_listen_ip: a local IP address to bind the server to. Default value: 127.0.0.1. Recommended to leave at this value, only change it if you plan on connecting external backends from other machines in your LAN.
debug: print out various debug information about incoming messages and events. Can be true or false.

The following fields in the config.json are specific to the Twitch internal backend:

tw_backend_enabled: whether or not the Twitch internal backend is turned on.
tw_nickname: your own Twitch nickname. You need to use the same nickname as the one you used for obtaining the token (see ("Connecting Twitch internal backend" section below).
tw_channel: the Twitch channel that you want to receive messages from (needs to start with #).

The following fields in the config.json are specific to the YouTube internal backend:

yt_backend_enabled: whether or not the YouTube internal backend is turned on.
yt_api_root_url: the API proxy root URL. Recommended to leave at https://yt.lemnoslife.com/noKey for now. This is a temporary field, it is subject to removal once the backend switches to using OAuth 2.0.
yt_api_poll_timeout: the interval (in seconds) to poll the YouTube chat with. For now, it's recommended to set to at least 3 seconds to not overload the API proxy.
yt_channel: the YouTube channel that you want to receive messages from (needs to start with @). If only this field is specified and the channel has several live broadcasts running, the backend will fetch the first broadcast from the list.
yt_video_id: an optional field that allows you to pass a particular broadcast video ID to get the live chat messages from. Overrides the yt_channel field.

Example config.json file:

{
  "server_port": 60053,
  "server_listen_ip": "127.0.0.1",
  "debug": true,
  "tw_backend_enabled": true,
  "tw_nickname": "Suborg",
  "tw_channel": "#foxyshadow",
  "yt_backend_enabled": true,
  "yt_api_root_url": "https://yt.lemnoslife.com/noKey",
  "yt_api_poll_timeout": 3,
  "yt_channel": "@foxyshadow"
}

Connecting Twitch internal backend

The following procedure needs to be only done once:

Go to the Twitch Chat OAuth Password Generator website.
Login with your own Twitch account and give it the access.
Copy the token string that starts with oauth: and paste it into the twitch-token.txt file in the auth subdirectory in the application directory.
Save the file. The Twitch connection should now work.
If the connection doesn't work, make sure that the tw_backend_enabled field is set to true in config.json and that you used the same Twitch nickname to get the token as the one specified in the tw_nickname field.

Connecting YouTube internal backend

As of now, nothing needs to be done except setting the yt_backend_enabled field to true in config.json, but this section will change soon once the backend moves to OAuth 2.0.

Customization

Being a Web-based application, the StreamGoose frontend provides vast customization capabilities on both the layout and client-side logic sides.

Styling

The webroot/style.css is the main point of customization. Here is a short (and vastly incomplete) reference on the CSS selectors and classes that you can operate on:

#msglist > div: an individual message block
.icon: user avatar (e.g. from YouTube backend)
.username: username text
.msg: message text
.timestamp: message timestamp (hidden by default)
.tw: a Twitch message (applies to the whole message block, as well as all the subsequent classes listed here)
.yt: a YouTube message
.me: third-party action
.broadcaster: a message that comes from the Twitch/YouTube channel owner
.moderator: a message that comes from a Twitch/YouTube chat moderator
.sponsor: a message that comes from a YouTube chat sponsor
.verified: a message that comes from a YouTube verified account

Additionally, every Twitch badge is converted into a CSS class that can be applied to a message block, e.g.

.vip: a message that comes from a Twitch VIP chat user
.subscriber: a message that comes from a Twitch subscriber
.admin: a message that comes from a Twitch admin

and so on. You can find a full list of badges and their IDs by issuing a GET request to https://api.twitchinsights.net/v1/badges/global URL and seeing the setID field.

Regex-based message highlighting

The webroot/custom.js allows you to pass custom JavaScript, including the predefined messageHighlightRules JS object. The keys in this object are regular expression strings, the values are the colors to apply to the messages where these expressions match. The matching is always case-insensitive and Unicode-aware. For the color values, you can use whatever CSS supports: #hex notation, symbolic names, rgba() or hsla() notation.

A canonical example is for matching alternative words in addition to the nickname:

var messageHighlightRules = {
  "(^|\\s)(suborg|luxferre|lux)(\,|;|:|\\s|$)": "#ff0000",
  "(^|\\s)(foxy|foxyshadow|fox)(\,|;|:|\\s|$)": "orange",
  "(^|\\s)(bluepanda|bluebear|sinipandoo)(\,|;|:|\\s|$)": "blue"
}

Each regular expression captures: the string start or a whitespace character, then one of the nickname alternatives, then either a comma, a semicolon, a colon or another whitespace character or the end of the string. If the message text matches the first expression, it is colored in red, if it matches the second one, it is colored in orange, if it matches the third one, it is colored in blue.

Backend implementation status

Twitch (internal)

Mostly stable. Some edge cases still could be found and further investigated.

Most chat badges are still ignored when rendering the messages.

YouTube (internal)

Currently, the YouTube backend doesn't have OAuth 2.0 flow implemented as of yet. A single API key would consume the quota too quickly, so, right now, the backend uses an unofficial Youtube API proxy which can and sometimes does have downtimes.

Rewriting the backend to support OAuth is one of the highest priorities.

Additionally, YouTube does not have its own chat emote API as of now, so the emote set is hardcoded in the /webroot/img directory and can become outdated over time.

FAQ

What were the reasons for starting this project?

In the game livestreaming community, there is some demand for a saner and more flexible alternative to the commercial live chat aggregation solutions, such as RutonyChat. In fact, witnessing several streamers struggling with some RutonyChat issues was the main trigger to start the project. On top of that, once the main functionality is complete, there are some plans to extend StreamGoose to support more things beyond just chat aggregation.

Why is it called StreamGoose?

This name was chosen as a winner of a poll offering 10 various names suggested by the members of two Discord servers with a lot of small-scale game livestreamers. It was one of the draft names too, given that, at the very beginning of the project, the broker server was written in C and based on the Mongoose library.

Additionally, this name turned out to be pretty unique — at the time of publishing this repo, no other desktop application with such a name existed.

How can I contribute?

Everyone is free to send their suggestions and patches. Once you get a Codeberg account, you can participate here in various ways:

Discuss, report and propose things at the issue tracker.
Create feature branches and merge requests if you know how to implement a particular feature in the Go-based StreamGoose core.
Write entirely new external independent backends in the programming languages of your choice (please place them into the /external directory of this repo). As a bonus, if you write a backend in Go and observe the same internal ChatMessage structure, we can even consider turning it into a Goroutine and include internally into the next StreamGoose version, provided it is already tested and stable enough.

The resulting binary size is around 6 megabytes or even more, is it normal?

Yes, due to the way Go links its networking client and server code and everything else statically (where possible). On Linux, you may try using GCC Go instead to achieve fully dynamic linking. Keep in mind that GCC Go is not fully supported yet, and if you come to our issue tracker with an issue that's only reproducible with GCC Go, it's less likely to be resolved.

Beyond Twitch and YouTube, are there any other livestreaming platforms that you plan to support internally?

Sure! For instance, one of the next close targets will be Owncast self-hosted livestreaming solution and its rather simple chat API. The problem with adding a new platform support, as always, is finding enough time to study the API and enough people that would find this support valuable. For example, with enough demand, Kick and Trovo support are also likely to be added.

Besides Twitch, YouTube and Owncast, however, your best bet would be to create an external backend for the platform you're interested in, and then incorporate it into this repo in the /external directory. When the backend is mature enough, we can always sit and think about how to convert it into an internal one to be distributed inside the StreamGoose binary.

Credits

Architectural design by Luxferre and members of @foxyshadow's Discord server.

Initial reference implementation by Luxferre.

Noto Color Emoji font by Google, distributed under the Open Font License.

All files, except the bundled font and image assets, are released into the public domain with no warranties.

17 KiB Raw Blame History

StreamGoose: an open-source livestreaming chat aggregator and widget helper