Like HTTP live streaming, but with IPFS which is maybe better?
This project is started by @ASoTNetworks and @darkdrgn2k to stream videos over IPFS, which overlapped with the need to live stream the Our Networks 2018 conference in Toronto. We will document here the components and processes necessary to run live streams throughout the conference and archive the video assets on the IPFS network, that is suitable for a small conference with an audience size of less than 100 people.
Here is a presentation and demo video about this project, recorded from the IPFS All Hands: November 26, 2018. Slides are available here.
We will record live streams in 720p (3 Mbps) and archive mp4 of each talk in 1080p.
Microphones
|
(xlr / wireless)
|
v
+------------------+
| Audio Mixer | --(3.5 mm audio)----+
+------------------+ |
v
+------------------+ +---------------+ +---------------------------+
| HD Video Camera | --(hdmi)--> | Elgato HD60 | --(usb)--> | Laptop running OBS Studio |
+------------------+ +---------------+ | ↳ Streams to RTMP server |
| for HTTP & IPFS streams |
+------------------+ +---------------+ | ↳ Records mp4 files |
| Presenter Laptop | --(hdmi)--> | AVerMedia LGP | --(usb)--> | for local archiving |
+------------------+ +---------------+ +---------------------------+
| | |
(hdmi pass-through) (usb) (ethernet)
| | |
v v v
Projector 1 TB HDD Gigabit Internet
The laptop is the control centre. It has two USB capture cards, connected to separate USB buses (e.g. if it has a USB2 and USB3 interface) if possible to avoid bandwidth issues. These will be the video and audio inputs. The capture cards are of two different brands because cards like the Elgato have problems when running two in parallel. At least one card should take a 3.5 mm audio input so we can mix the audio into the stream via the audio mixer.
The laptop runs the following software:
For Yggdrasil, you should compile at tag v0.3.2
, and when streaming, run it with the
configurations that will be downloaded from the streaming server at a later step.
OBS Studio is used throughout the conference to toggle between the two video feeds (i.e.
the slides and the presenter video). Using the Start Streaming
function in OBS Studio,
the stream is published at 720p to a RTMP server we will set up in the next step. Using
the Start Recording
function in OBS Studio, the operator will also record each talk as
a separate 1080p mp4 file to the external hard disk to be published after the event.
OBS Studio Website Embedded Viewer with
Source Video Player IPFS Client
| ^ ^ ^ ^
(rtmp-publish) | | | |
| (http) (ipfs) (http) (ipfs)
v | | | |
+-------------------+ +---------------------+ +---------------------+
| rtmp-server | | ipfs-server | | ipfs-mirror |
| ↳ nginx-rtmp | | ↳ ipfs with pubsub | | ↳ ipfs with pubsub |
| ↳ openvpn |<--(rtmp-pull)--| ↳ ipfs-http gateway |<--(ipfs)--| ↳ ipfs-http gateway |
| ↳ yggdrasil | | ↳ ffmpeg | |- - - - - - - - - - -|
|- - - - - - - - - -| |- - - - - - - - - - -| | Pins IPFS hashes |
| Runs RTMP server | | Encodes HLS ts+m3u8 | | learnt from IPNS id |
| publishable from | | pins on IPFS and | | of ipfs-server |
| authenticated IPs | | publishes to IPNS | +---------------------+
+-------------------+ +---------------------+
|
(rtmp-pull/push)
|
v
Other Streaming Services
The on-premise laptop running OBS Studio pushes to the rtmp-server
, which through
IP-pinning of the OpenVPN or Yggdrasil-generated IP address will allow only that device to
publish. The ipfs-server
pulls that RTMP stream, encodes ts chunks in a live m3u8 file using
ffmpeg, then IPFS adds and pins those files and uses IPNS to publish the m3u8 to its node
ID. The built-in ipfs-http gateway allow those content to be accessed via HTTP, which is
what the embedded player on the website will use. However, viewers running a IPFS client
(with pubsub enabled) can directly view the streams over IPFS. Optionally, we can run
one or more ipfs-mirror
servers that pin the live streaming content and run additional
gateways.
All the servers described above are provisioned using Terraform on Digital Ocean. In addition, the RTMP stream can be consumed by other services to provide a parallel stream that does not involve IPFS.
We will be using the following tools and services:
The following steps assume you have a Digital Ocean account and the above listed software installed on your local machine, which can be the same device running OBS Studio.
Clone this repository and work from the terraform
directory:
git clone https://github.com/tomeshnet/ipfs-live-streaming.git
cd ipfs-live-streaming/terraform
From your domain name registrar, point name servers to Digital Ocean's name servers:
ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com
Then store the domain name in your local environment:
echo -n YOUR_DOMAIN_NAME > .keys/domain_name
Set an email address to use as contact email for Let's Encrypt:
echo -n YOUR_EMAIL_ADDRESS > .keys/email_address
DEVELOPER TIP Set dryrun to use Let's Encrypt staging servers for testing
echo -n true > .keys/dryrun
Obtain a read-write access token from your Digital Ocean account's API
tab, then store
it in your local environment:
echo -n YOUR_DIGITAL_OCEAN_ACCESS_TOKEN > .keys/do_token
Generate RSA keys to access your Digital Ocean VMs:
ssh-keygen -t rsa -f .keys/id_rsa
Add the SSH key to your Digital Ocean account under Settings > Security
, then copy the
SSH fingerprint to your local environment:
echo -n YOUR_SSH_FINGERPRINT > .keys/ssh_fingerprint
Download Terraform, add it to your path. On Linux it would look something like this:
https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
unzip terraform_0.11.7_linux_amd64.zip
mv terraform /usr/bin
Then run initialization from our terraform
working directory:
terraform init
Provision the streaming servers by running:
terraform apply
By default, this will create one instance of each server type. You may choose to create
multiple instances of ipfs-mirror
by overriding the mirror
variable.
You may also choose to include an external HTTP (non-IPFS) stream source by adding one or more URLs to a m3u8 playlist.
For example:
terraform apply \
-var "mirror=2" \
-var "m3u8_http_urls=\'https://HLS_SOURCE_0/live.m3u8\',\'https://HLS_SOURCE_1/live.m3u8\'"
From your browser, login to your Digital Ocean dashboard and find your new VMs tagged
with ipfs-live-streaming
.
You will find a couple new files in your .keys
folder:
client.conf (for OpenVPN on Linux)
client.ovpn (for OpenVPN on MacOS and Windows)
yggdrasil.conf (for Yggdrasil)
To authenticate using OpenVPN, connect with your OpenVPN client using client.conf
or
client.ovpn
, then publish your OBS Studio stream to:
rtmp://10.10.10.1:1935/live/stream1
To authenticate using Yggdrasil, start it with yggdrasil.conf
and note the last line of
output like this:
sudo yggdrasil --useconf < ./keys/yggdrasil.conf
...
2018/12/14 15:16:22 Connected: 203:4bb0:9ff1:2312:e7f3:b8c4:852:[email protected] source 192.168.1.173
Then publish your OBS Studio stream to the IPv6:
rtmp://[203:4bb0:9ff1:2312:e7f3:b8c4:852:a8b1]:1935/live/stream1
When your streaming session is done, you can stop OpenVPN or Yggdrasil and destroy the servers with:
terraform destroy
The following streams become publicly available about 30 seconds after you start publishing to the RTMP server:
Stream | URL |
---|---|
RTMP stream | rtmp://rtmp-server.YOUR_DOMAIN_NAME/live |
HLS stream (origin) | https://YOUR_DOMAIN_NAME/live.m3u8 |
HLS stream (mirror-N) | https://ipfs-mirror-N.YOUR_DOMAIN_NAME/live.m3u8 |
IPNS HLS stream (origin) | https://ipfs-gateway.YOUR_DOMAIN_NAME/ipns/IPNS_ID (disabled) |
IPNS HLS stream (mirror-N) | https://ipfs-gateway-N.YOUR_DOMAIN_NAME/ipns/IPNS_ID (disabled) |
The origin ipfs-server
and each ipfs-mirror
also host an embedded video player, publicly available at:
Site | URL |
---|---|
Video player (origin) | https://YOUR_DOMAIN_NAME |
Video player (mirror-N) | https://ipfs-mirror-N.YOUR_DOMAIN_NAME |
The embedded video player is tested on common desktop and mobile browsers, and support the following optional URL query parameters:
Parameter | Description |
---|---|
gw |
Set custom IPFS gateway |
m3u8 |
Set m3u8 file URL to override IPFS live stream |
vod |
Set IPFS content hash of mp4 file to play IPFS on-demand video stream |
from |
Set IPFS content hash or timecode to start video playback from |
The video player uses code from Video.js, graphics from ipfs/artwork, and loading animation from jxnblk/loading.