luxferre/StreamGoose
1
0
Fork 0
Code Actions Activity
Files
842452f69020e7ef820ba525e636a8c78002d17d
StreamGoose/README.md
T

342 lines
21 KiB
Markdown
Raw Blame History

# StreamGoose: an open-source livestreaming chat aggregator and widget helper
StreamGoose (aka StreamGoOSE, Streamer's Go-based Open Source Enhancer) is an open tool to solve the problem of aggregating several live chats to display on a livestream.
As of now, most of the StreamGoose codebase is written in Go 1.23.1. The Web part is written in HTML5, CSS3 and JavaScript using the ES2015 standard.
## Current features
The status of StreamGoose is currently **alpha**. Use at your own risk.
Nevertheless, the following features are already implemented:
- Twitch, YouTube, Owncast and Kick chat aggregation (see "Backend implementation status" section)
- Handling Twitch emotes
- Automated Twitch chat badges output (global and channel-specific)
- Handling YouTube non-Unicode emotes (see "Backend implementation status" section)
- Handling Kick emotes
- Message styling based on Twitch and YouTube roles
- Twitch username styling based on color preference
- Message highlighting based on regular expressions
- Third-party action (`/me`) command handling
- YouTube avatar icon support
- Twitch system message support (needs more testing)
- [Noto Color Emoji](https://fonts.google.com/noto/specimen/Noto+Color+Emoji) font bundled for universal Unicode emoji support
## Planned features
- Reward redemption message support for Twitch
- "Deleted message" support for YouTube
- Badge support for Kick
- Internal chat ranking system by username
- Backend-independent support for oldschool plaintext emoticons ;-)
- Starting window geometry customization via config (now doable via custom JS)
- ...
## Architecture
StreamGoose consists of three components: a set of backends, a message broker server and a Web-based frontend.
### Message broker server
This is the main StreamGoose component. It provides three main functions:
- accepting new messages from backends via external (`POST /new`) and internal (`message_transport` Go channel) ways;
- providing the unified message queue for the frontend side consumption;
- serving static files that make up the frontend (`index.html`, `script.js`, `style.css` and images) over the plain HTTP protocol.
Additionally, the server binary also launches the backends it knows about and opens a simple GUI window to show the Web content locally, feeding its unified output into the JS side via server-sent events. The latter part became possible thanks to the [webview_go](https://pkg.go.dev/centrifuge.hectabit.org/HectaBit/webview_go) library.
### Backends
StreamGoose can operate with two kinds of backends: internal and external.
An **internal** StreamGoose backend is a Go program embedded as a goroutine into the main binary. It communicates with the message broker by sending messages to the `message_transport` Go channel. The messages are defined using this Go structure:
```go
type ChatMessage struct {
Id uint64 `json:"id"` /* message id (auto-filled) */
NativeId string `json:"native_id"` /* message id native to the platform */
AuthorId string `json:"author_id"` /* author id native to the platform */
Type string `json:"type"` /* message type string */
Timestamp string `json:"timestamp"` /* timestamp string (formatted) */
Username string `json:"username"` /* author username (as displayed) */
Text string `json:"text"` /* message text */
Icon string `json:"icon"` /* optional message icon URI */
}
```
The `Id` field is populated automatically whenever a message enters the broker. For others, see below for the detailed description of the corresponding JSON fields in the external backends.
An **external** StreamGoose backend is a program (written in any network-enabled programming language) that can make a POST request to the corresponding message broker URL (`http://[broker_host]:60053/new` endpoint). The request body shall be a JSON object with the following fields (all passed as strings):
- `native_id`: an ID string assigned to the message by the streaming platform itself (Twitch, YouTube etc). Used to further handling deleted messages.
- `author_id`: an ID string assigned to the message author by the streaming platform itself. Used to further handle bans and subsequent message deletions.
- `type`: message type, consisting of `^`-delimited classes (e.g. `yt^moderator` or `tw^vip^color#A1B2C3` etc, see below in the "Customization" section of this README). This field defines how the message will be styled when being rendered on the frontend. There are two special message types, `delmsg` and `deluser`, that require `native_id` and `author_id` fields respectively to be set in order to perform message deletion on the frontend.
- `timestamp`: an ISO 8601 **UTC** timestamp string denoting when the message was sent. The only recommended format (in strftime syntax) is `%Y-%m-%dT%H:%M:%SZ`.
- `username`: the message author name that should be displayed in the chat.
- `text`: the message text that should be displayed in the chat.
- `icon`: an optional icon URL passed to the chat if the message author has a personal user profile picture (avatar). This field can be left empty but should be included in the request anyway.
Whenever a new JSON message is shaped with the fields above, it shall be sent to the broker at the `/new` endpoint with the HTTP POST method. A `{"status":1}` response will be sent back if the request is successful.
As of now, StreamGoose provides three reference backends: for Twitch, for YouTube and for Owncast, all of them internal and written in Go. See "Backend implementation status" section for current notes and caveats.
### Frontend
The StreamGoose frontend is a Web-based (HTML5) application served by the message broker server. It is opened in the built-in WebView-based GUI and provides a constantly updated aggregated live chat display, message type dependent styling, emote handling, custom highlighting and so on. The users are free and even encouraged to modify the web assets located in the `/webroot` directory to fit the chat to their own visual preferences.
## Building and installation
### Linux and macOS
You need to have Go 1.23.1 and above and a POSIX-compatible "make".
For Linux, you also need to have the `webkit2gtk-4.1` dev package installed.
Just run `make` and get all the resulting files in the `out` directory.
### Windows x64 (native)
You need to have Go 1.23.1 and above and a POSIX-compatible "make" in your %PATH%.
Just run `make win64` and get all the resulting files in the `out` directory.
### Windows x64 (cross-compiling from Linux)
You need to have Go 1.23.1 and above and a POSIX-compatible "make".
1. Install the `mingw-w64` compiler package.
2. Run `make win64-cross` and get all the resulting files in the `out` directory.
If the compiler complains about the `EventToken.h` file when building `webview_go` dependency, locate the include files and symlink it to the real `eventtoken.h` like this (as root):
```
# ln -s /usr/x86_64-w64-mingw32/include/eventtoken.h /usr/x86_64-w64-mingw32/include/EventToken.h
```
Then rerun the command from step 2.
**Note**: When testing the cross-compiled build under Wine, you must have Microsoft Edge WebView2 Runtime installed in order for the GUI to work at all. You can [download](https://developer.microsoft.com/en-us/microsoft-edge/webview2) this engine directly from the Microsoft website (select "Evergreen Standalone Installer") and install it into your Wine profile. Be sure to enable at least Windows 7 emulation in winecfg.
### Headless builds
In case you don't want to build the GUI, you can run `make headless` or `make win64-headless` depending on the platform. In this case, a binary will be built where you'll be required to point any JS-enabled Web browser of your choice to `http://127.0.0.1:60053` to see the chat window. This can be useful when building StreamGoose on more exotic platforms not supported by the `webview_go` library.
Note that the `make win64-headless` command can be run on any supported OS, all cross-compilation is done transparently in this case.
## Usage
Enter the distribution directory and run the `./streamgoose` or `streamgoose.exe` binary. It should run with the default `config.json` file, start the server, spawn the internal backends and open the UI window. Alternatively, you can pass another JSON file as the first command line parameter.
### Configuration
StreamGoose is mostly configured through the `config.json` file distributed in the same directory as the main binary.
The following fields in the `config.json` are backend-agnostic:
- `server_port`: a local port to run the HTTP REST server on. Default value: `60053` (for "GOOSE").
- `server_listen_ip`: a local IP address to bind the server to. Default value: `127.0.0.1`. Recommended to leave at this value, only change it if you plan on connecting external backends from other machines in your LAN.
- `debug`: print out various debug information about incoming messages and events. Can be `true` or `false`.
- `useragent`: the value of the `User-Agent` HTTP header used by the backends with unofficial APIs, like Kick and YouTube.
The following fields in the `config.json` are specific to the Twitch internal backend:
- `tw_backend_enabled`: whether or not the Twitch internal backend is turned on.
- `tw_nickname`: **your own** Twitch nickname. You need to use the same nickname as the one you used for obtaining the token (see ("Connecting Twitch internal backend" section below).
- `tw_channel`: the Twitch channel that you want to receive messages from (needs to start with `#`).
The following fields in the `config.json` are specific to the YouTube internal backend:
- `yt_backend_enabled`: whether or not the YouTube internal backend is turned on.
- `yt_api_poll_timeout`: the interval (in seconds) to poll the YouTube chat with.
- `yt_channel`: the YouTube channel that you want to receive messages from (needs to start with `@`). If only this field is specified and the channel has several live broadcasts running, the backend will fetch the first broadcast from the list.
- `yt_video_id`: an optional field that allows you to pass a particular broadcast video ID to get the live chat messages from. Overrides the `yt_channel` field.
The following fields in the `config.json` are specific to the Owncast internal backend:
- `oc_backend_enabled`: whether or not the Owncast internal backend is turned on.
- `oc_server`: your Owncast server root URL.
- `oc_api_poll_timeout`: the interval (in seconds) to poll the Owncast chat with. Recommended to set to at least 2 seconds.
The following fields in the `config.json` are specific to the Kick internal backend:
- `kk_backend_enabled`: whether or not the Owncast internal backend is turned on.
- `kk_channel`: your Kick channel name.
- `kk_api_poll_timeout`: the interval (in seconds) to poll the Kick chat with. Recommended to set to at least 2 seconds.
Example `config.json` file:
```json
{
"server_port": 60053,
"server_listen_ip": "127.0.0.1",
"debug": true,
"useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0",
"tw_backend_enabled": true,
"tw_nickname": "Suborg",
"tw_channel": "#foxyshadow",
"yt_backend_enabled": true,
"yt_api_poll_timeout": 1,
"yt_channel": "@foxyshadow",
"oc_backend_enabled": true,
"oc_server": "https://streams.luxferre.top",
"oc_api_poll_timeout": 2,
"kk_backend_enabled": true,
"kk_channel": "foxyshadow",
"kk_api_poll_timeout": 2
}
```
### Connecting Twitch internal backend
The following procedure needs to be only done once per ~2 months:
1. Go to the Twitch Chat OAuth Password Generator website using [this link](https://id.twitch.tv/oauth2/authorize?response_type=token&client_id=q6batx0epp608isickayubi39itsckt&redirect_uri=https://twitchapps.com/tmi/&scope=chat:read+chat:edit+channel:read:redemptions+channel:moderate+whispers:read+whispers:edit+channel_editor).
2. Login with your own Twitch account and give it the access.
3. Copy the token string that starts with `oauth:` and paste it into the `twitch-token.txt` file in the `auth` subdirectory in the application directory (note that it is different from `twitch-cid.txt` that already is present in this subdirectory).
4. Save the file. The Twitch connection should now work.
5. If the connection doesn't work, make sure that the `tw_backend_enabled` field is set to `true` in `config.json` and that you used the same Twitch nickname to get the token as the one specified in the `tw_nickname` field.
### Connecting YouTube internal backend
As of now, nothing needs to be done except setting the `yt_backend_enabled` field to `true` in `config.json`, as well as setting `yt_api_poll_timeout` and the target channel.
### Connecting Owncast internal backend
The following procedure needs to be only done once:
1. Go to the admin panel of your Owncast server. Open the "Integrations" - "Access Tokens" menu item.
2. Create a new access token with the "Can perform administrative actions such as moderation, get server statuses, etc" permission.
3. Copy the token string generated in the admin panel and paste it into the `oc-token.txt` file in the `auth` subdirectory in the application directory.
4. Save the file. The Owncast connection should now work and fetch messages whenever you start a livestream.
5. If the connection doesn't work, make sure that the `oc_backend_enabled` field is set to `true` in `config.json` and you didn't revoke the token in the admin panel.
### Connecting Kick internal backend
As of now, nothing needs to be done except setting the `kk_backend_enabled` field to `true` in `config.json`, but this section might be subject to change once Kick deploys and enforces any official APIs.
### Capturing the chat to display on your livestream
Normally, you can just capture the StreamGoose window contents with whatever streaming software you're using. If using OBS, you can alternatively select a "Browser source" and point it to the `http://127.0.0.1:60053` URL. In this case, you can also use a headless build of StreamGoose.
## Customization
Being a Web-based application, the StreamGoose frontend provides vast customization capabilities on both the layout and client-side logic sides.
### Styling
The `webroot/style.css` is the main point of customization. Here is a short (and vastly incomplete) reference on the CSS selectors and classes that you can operate on:
- `#msglist > div`: an individual message block
- `.icon`: user avatar (e.g. from YouTube backend)
- `.username`: username text
- `.msg`: message text
- `.timestamp`: message timestamp (hidden by default)
- `.tw`: a Twitch message (applies to the whole message block, as well as **all the subsequent classes** listed here)
- `.yt`: a YouTube message
- `.kk`: a Kick message
- `.oc`: an Owncast message
- `.me`: third-party action
- `.broadcaster`: a message that comes from the Twitch/YouTube channel owner
- `.moderator`: a message that comes from a Twitch/YouTube chat moderator
- `.sponsor`: a message that comes from a YouTube chat sponsor
- `.verified`: a message that comes from a YouTube verified account
- `.authenticated`: a message that comes from an Owncast authenticated user
- `.bot`: a message that comes from an Owncast bot
Additionally, every Twitch badge is converted into a CSS class that can be applied to a message block, e.g.
- `.vip`: a message that comes from a Twitch VIP chat user
- `.subscriber`: a message that comes from a Twitch subscriber
- `.admin`: a message that comes from a Twitch admin
and so on. You can find a full list of badges and their IDs by issuing a GET request to `https://api.twitchinsights.net/v1/badges/global` URL and seeing the `setID` field.
### Regex-based message highlighting
The `webroot/custom.js` allows you to pass custom JavaScript, including the predefined `messageHighlightRules` JS object. The keys in this object are regular expression strings, the values are the colors to apply to the messages where these expressions match. The matching is always case-insensitive and Unicode-aware. For the color values, you can use whatever CSS supports: `#hex` notation, symbolic names, `rgba()` or `hsla()` notation.
A canonical example is for matching alternative words in addition to the nickname:
```js
var messageHighlightRules = {
"(^|\\s)(suborg|luxferre|lux)(\,|;|:|\\s|$)": "#ff0000",
"(^|\\s)(foxy|foxyshadow|fox)(\,|;|:|\\s|$)": "orange",
"(^|\\s)(bluepanda|bluebear|sinipandoo)(\,|;|:|\\s|$)": "blue"
}
```
Each regular expression captures: the string start or a whitespace character, then one of the nickname alternatives, then either a comma, a semicolon, a colon or another whitespace character or the end of the string. If the message text matches the first expression, it is colored in red, if it matches the second one, it is colored in orange, if it matches the third one, it is colored in blue.
## Backend implementation status
### Twitch (internal)
Mostly stable. Some edge cases still could be found and further investigated.
### YouTube (internal)
After thorough investigation with various methods (official API via keys and OAuth, unofficial YouTube Data API proxy), a decision was made to switch to the Innertube chat API. It seems to be the most stable and unsrestricted for the time being.
Additionally, YouTube does not have its own chat emote API as of now, so the emote set is hardcoded in the `/webroot/img` directory and can become outdated over time. There is a plan to implement dynamic emore loading from the data inside the message text runs retrieved via the Innertube API.
### Owncast (internal)
Work in progress. Needs additional testing. Basic messages work as expected.
### Kick (internal)
Work in progress. Relies on an API that hasn't been officially released yet and an external library to bypass CloudFlare checks. Needs additional testing. Basic messages work as expected.
## FAQ
### What were the reasons for starting this project?
In the game livestreaming community, there is some demand for a saner and more flexible alternative to the commercial live chat aggregation solutions, such as RutonyChat. In fact, witnessing several streamers struggling with some RutonyChat issues was the main trigger to start the project. On top of that, once the main functionality is complete, there are some plans to extend StreamGoose to support more things beyond just chat aggregation.
### Why is it called StreamGoose?
This name was chosen as a winner of a poll offering 10 various names suggested by the members of two Discord servers with a lot of small-scale game livestreamers. It was one of the draft names too, given that, at the very beginning of the project, the broker server was written in C and based on the [Mongoose](https://mongoose.ws) library.
Additionally, this name turned out to be pretty unique — at the time of publishing this repo, no other desktop application with such a name existed.
### How can I contribute?
Everyone is free to send their suggestions and patches. Once you get a Codeberg account, you can participate here in various ways:
1. Discuss, report and propose things at the issue tracker.
2. Create feature branches and merge requests if you know how to implement a particular feature in the Go-based StreamGoose core.
3. Write entirely new external independent backends in the programming languages of your choice (please place them into the `/external` directory of this repo). As a bonus, if you write a backend in Go and observe the same internal `ChatMessage` structure, we can even consider turning it into a Goroutine and include internally into the next StreamGoose version, provided it is already tested and stable enough.
### The resulting binary size is around 6 megabytes or even more, is it normal?
Yes, due to the way Go links its networking client and server code and everything else statically (where possible). On Linux, you may try using GCC Go instead to achieve fully dynamic linking. Keep in mind that GCC Go is not fully supported yet, and if you come to our issue tracker with an issue that's only reproducible with GCC Go, it's less likely to be resolved.
### Beyond Twitch, YouTube, Owncast and Kick, are there any other livestreaming platforms that you plan to support internally?
Sure, anything can happen. The problem with adding a new platform support, as always, is finding enough time to study the API and enough people that would find this support valuable. For example, with enough demand, Trovo support is also likely to be added, however, it would require significantly more time and effort that could be leveraged for improving e.g. YouTube backend that would have significantly larger impact.
Besides the main four, anyway, your best bet would be to create an **external** backend for the platform you're interested in (again, you could use any programming language you want), and then incorporate it into this repo in the `/external` directory. When the backend is mature enough, we can always sit and think about how to convert it into an internal one to be distributed inside the StreamGoose binary.
### Why is there a `twitch-cid.txt` file in the `auth/` directory?
This is the Twitch client ID corresponding to the [Twitch Chat OAuth Password Generator](https://twitchapps.com/tmi/) website. If you generate a Twitch token through it, you'll need this ID for external Twitch API such as badges to work correctly.
Twitch client IDs are not considered secrets and can be included in the source code.
## Credits
Architectural design by [Luxferre](https://luxferre.top) and members of [@foxyshadow](https://twitch.tv/foxyshadow)'s Discord server.
Initial reference implementation by Luxferre.
Noto Color Emoji font by Google, distributed under the [Open Font License](https://openfontlicense.org).
All files, except the bundled font and image assets, are released into the public domain with no warranties.
View Git Blame Copy Permalink