Compare commits

...

19 Commits

Author SHA1 Message Date
Marco Piovanello
e0ebe9a995
Copilot is thinking...
Do we really need an AI to generate a commit description??
2026-01-21 10:32:30 +01:00
Marco Piovanello
8c06485880
fixed authentication middleware 2025-09-01 18:31:01 +02:00
Marco Piovanello
ccb6bbe3e6
fixed auth middleware 2025-08-31 13:16:36 +02:00
marcobaobao
9ca7bb9377
updated twitch dialog component labels 2025-08-28 20:30:36 +02:00
marcobaobao
bce696fc67
fixed version string 2025-08-28 14:42:18 +02:00
marcobaobao
22caf8899b
added twitch frontend components 2025-08-28 14:40:04 +02:00
marcobaobao
2a11f64935
default value in twitch config 2025-08-27 10:10:54 +02:00
Marco Piovanello
f4a0f688af
Feat twitch livestreams (#334)
* backend code

* fixed twitch authentication
2025-08-25 12:54:16 +02:00
Marco Piovanello
14a03d6a77
Prevent RCEs with crafted inputs 2025-07-23 10:21:34 +02:00
Marco Piovanello
8a73079fad
Update Dockerfile 2025-04-13 20:13:59 +02:00
marcobaobao
f578f44cfd
refactor: prevent multiple slashes 2025-03-30 10:29:13 +02:00
marcobaobao
cbe16c5c6c
refactoring: readded abort controller to httpClient.ts 2025-03-30 10:21:19 +02:00
marcobaobao
3cebaf7f61
refactor: extra slashes prevention 2025-03-30 10:17:30 +02:00
Marco Piovanello
2d2cb1dc3a
Update README.md 2025-03-30 09:54:27 +02:00
Marco Piovanello
43bcc40907
293 tiny gui improvement (#296)
* clicking on the speed dial will open download dialog

* refactor: prevent multiple slashes
2025-03-29 21:27:28 +01:00
Marco Piovanello
2af27e51be
Chore dockerfile refactor (#287)
* removed yt-dlp alpine package

* use python3-alpine base image
2025-03-22 16:17:25 +01:00
Marco Piovanello
8c18242aaf
removed yt-dlp alpine package (#286) 2025-03-22 15:27:48 +01:00
Marco Piovanello
66bebb2529
Update README.md 2025-03-17 11:23:29 +01:00
Marco Piovanello
e223e030ac
restrict user with a whitelist (#282) 2025-03-17 11:13:20 +01:00
23 changed files with 829 additions and 357 deletions

1
.gitignore vendored
View File

@ -29,3 +29,4 @@ frontend/.yarn/install-state.gz
livestreams.dat
.vite/deps
archive.txt
twitch-monitor.dat

View File

@ -24,11 +24,12 @@ COPY --from=ui /usr/src/yt-dlp-webui/frontend /usr/src/yt-dlp-webui/frontend
RUN CGO_ENABLED=0 GOOS=linux go build -o yt-dlp-webui
# -----------------------------------------------------------------------------
# dependencies ----------------------------------------------------------------
FROM alpine:edge
# Runtime ---------------------------------------------------------------------
FROM python:3.13.2-alpine3.21
RUN apk update && \
apk add ffmpeg yt-dlp ca-certificates curl wget psmisc
apk add ffmpeg ca-certificates curl wget gnutls --no-cache && \
pip install "yt-dlp[default,curl-cffi,mutagen,pycryptodomex,phantomjs,secretstorage]"
VOLUME /downloads /config
@ -39,4 +40,4 @@ COPY --from=build /usr/src/yt-dlp-webui/yt-dlp-webui /app
ENV JWT_SECRET=secret
EXPOSE 3033
ENTRYPOINT [ "./yt-dlp-webui" , "--out", "/downloads", "--conf", "/config/config.yml", "--db", "/config/local.db" ]
ENTRYPOINT [ "./yt-dlp-webui" , "--out", "/downloads", "--conf", "/config/config.yml", "--db", "/config/local.db" ]

315
README.md
View File

@ -1,312 +1,17 @@
> [!NOTE]
> A poll is up to decide the future of yt-dlp-web-ui frontend! If you're interested you can take part.
> https://github.com/marcopiovanello/yt-dlp-web-ui/discussions/223
## I'm migrating away from GitHub
# yt-dlp Web UI
This project is not, and will not, use AI generated code or any AI "powered" tool.
Fuck copilot, fuck openAI and fuck all the corporate greed shit that is stripping off our privacy, making our life worse and destryoing the planet.
A not so terrible web ui for yt-dlp.
GitHub (basically Microsoft) is training models with ass-level code, found in the majority of the public (and even) private repositories, to help us produce more ass-level code and making worse products.
High performance extendeable web ui and RPC server for yt-dlp with low impact on resources.
Maybe the funny guy with aviator sunglasses was right.
Created for the only purpose of *fetching* videos from my server/nas and monitor upcoming livestreams.
![jsilverhand](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQ-Y2l40hcgIwy6IDDdSYOfYZO35JfFWKH2iw&s)
**Docker images are available on [Docker Hub](https://hub.docker.com/r/marcobaobao/yt-dlp-webui) or [ghcr.io](https://github.com/marcopiovanello/yt-dlp-web-ui/pkgs/container/yt-dlp-web-ui)**.
### I'm still going to contribute in this project... but...
```sh
docker pull marcobaobao/yt-dlp-webui
```
```sh
# latest dev
docker pull ghcr.io/marcopiovanello/yt-dlp-web-ui:latest
```
We can meet here: [https://gitea.aidystopia.xyz](https://gitea.aidystopia.xyz/marco/yt-dlp-webui). I'm migrating everything to my presonal gitea.
See you there!
## Donate to yt-dlp-webui development
[PayPal](https://paypal.me/marcofw)
*Keeps the project alive!* 😃
## Community stuff
Feel free to join :)
[![Discord Banner](https://api.weblutions.com/discord/invite/3Sj9ZZHv/)](https://discord.gg/WRnVWr4y)
## Some screeshots
![image](https://github.com/user-attachments/assets/fc43a3fb-ecf9-449d-b5cb-5d5635020c00)
![image](https://github.com/user-attachments/assets/3210f6ac-0dd8-403c-b839-3c24ff7d7d00)
![image](https://github.com/user-attachments/assets/16450a40-cda6-4c8b-9d20-8ec36282f6ed)
## Video showcase
[app.webm](https://github.com/marcopiovanello/yt-dlp-web-ui/assets/35533749/91545bc4-233d-4dde-8504-27422cb26964)
## Settings
The currently avaible settings are:
- Server address
- Switch theme
- Extract audio
- Switch language
- Optional format selection
- Override the output filename
- Override the output path
- Pass custom yt-dlp arguments safely
- Download queue (limit concurrent downloads)
## Format selection
This feature is disabled by default as this intended to be used to retrieve the best quality automatically.
To enable it just go to the settings page and enable the **Enable video/audio formats selection** flag!
## [Docker](https://github.com/marcopiovanello/yt-dlp-web-ui/pkgs/container/yt-dlp-web-ui) run
```sh
docker pull marcobaobao/yt-dlp-webui
docker run -d -p 3033:3033 -v <your dir>:/downloads marcobaobao/yt-dlp-webui
```
Or with docker but building the container manually.
```sh
docker build -t yt-dlp-webui .
docker run -d -p 3033:3033 -v <your dir>:/downloads yt-dlp-webui
docker run -d -p 3033:3033 \
-v <your dir>:/downloads \
-v <your dir>:/config \ # optional
yt-dlp-webui
```
If you opt to add RPC authentication...
```sh
docker run -d \
-p 3033:3033 \
-e JWT_SECRET randomsecret
-v /path/to/downloads:/downloads \
-v /path/for/config:/config \ # optional
marcobaobao/yt-dlp-webui \
--auth \
--user your_username \
--pass your_pass
```
If you wish for limiting the download queue size...
e.g. limiting max 2 concurrent download.
```sh
docker run -d \
-p 3033:3033 \
-e JWT_SECRET randomsecret
-v /path/to/downloads:/downloads \
marcobaobao/yt-dlp-webui \
--qs 2
```
### Docker Compose
```yaml
services:
yt-dlp-webui:
image: marcobaobao/yt-dlp-webui
ports:
- 3033:3033
volumes:
- <your dir>:/downloads # replace <your dir> with a directory on your host system
healthcheck:
test: curl -f http://localhost:3033 || exit 1
restart: unless-stopped
```
### ⚡ One-Click Deploy
| Cloud Provider | Deploy Button |
|----------------|---------------|
| AWS | <a href="https://deploystack.io/deploy/marcopiovanello-yt-dlp-web-ui?provider=aws&language=cfn"><img src="https://raw.githubusercontent.com/deploystackio/deploy-templates/refs/heads/main/.assets/img/aws.svg" height="38"></a> |
| DigitalOcean | <a href="https://deploystack.io/deploy/marcopiovanello-yt-dlp-web-ui?provider=do&language=dop"><img src="https://raw.githubusercontent.com/deploystackio/deploy-templates/refs/heads/main/.assets/img/do.svg" height="38"></a> |
| Render | <a href="https://deploystack.io/deploy/marcopiovanello-yt-dlp-web-ui?provider=rnd&language=rnd"><img src="https://raw.githubusercontent.com/deploystackio/deploy-templates/refs/heads/main/.assets/img/rnd.svg" height="38"></a> |
<sub>Generated by <a href="https://deploystack.io/c/marcopiovanello-yt-dlp-web-ui" target="_blank">DeployStack.io</a></sub>
## [Prebuilt binaries](https://github.com/marcopiovanello/yt-dlp-web-ui/releases) installation
```sh
# download the latest release from the releases page
mv yt-dlp-webui_linux-[your_system_arch] /usr/local/bin/yt-dlp-webui
# /home/user/downloads as an example and yt-dlp in $PATH
yt-dlp-webui --out /home/user/downloads
# specifying yt-dlp path
yt-dlp-webui --out /home/user/downloads --driver /opt/soemdir/yt-dlp
# specifying using a config file
yt-dlp-webui --conf /home/user/.config/yt-dlp-webui.conf
```
### Arguments
```sh
Usage yt-dlp-webui:
-auth
Enable RPC authentication
-conf string
Config file path (default "./config.yml")
-db string
local database path (default "local.db")
-driver string
yt-dlp executable path (default "yt-dlp")
-fl
enable file based logging
-host string
Host where server will listen at (default "0.0.0.0")
-lf string
set log file location (default "yt-dlp-webui.log")
-out string
Where files will be saved (default ".")
-pass string
Password required for auth
-port int
Port where server will listen at (default 3033)
-qs int
Queue size (concurrent downloads) (default 2)
-session string
session file path (default ".")
-user string
Username required for auth
-web string
frontend web resources path
```
### Config file
By running `yt-dlp-webui` in standalone mode you have the ability to also specify a config file.
The config file **will overwrite what have been passed as cli argument**.
With Docker, inside the mounted `/conf` volume inside there must be a file named `config.yml`.
```yaml
# Simple configuration file for yt-dlp webui
---
# Host where server will listen at (default: "0.0.0.0")
#host: 0.0.0.0
# Port where server will listen at (default: 3033)
port: 8989
# Directory where downloaded files will be stored (default: ".")
downloadPath: /home/ren/archive
# [optional] Enable RPC authentication (requires username and password)
require_auth: true
username: my_username
password: my_random_secret
# [optional] The download queue size (default: logical cpu cores)
queue_size: 4 # min. 2
# [optional] Full path to the yt-dlp (default: "yt-dlp")
#downloaderPath: /usr/local/bin/yt-dlp
# [optional] Enable file based logging with rotation (default: false)
#enable_file_logging: false
# [optional] Directory where the log file will be stored (default: ".")
#log_path: .
# [optional] Directory where the session database file will be stored (default: ".")
#session_file_path: .
# [optional] Path where the sqlite database will be created/opened (default: "./local.db")
#local_database_path
# [optional] Path where a custom frontend will be loaded (instead of the embedded one)
#frontend_path: ./web/solid-frontend
```
### Systemd integration
By defining a service file in `/etc/systemd/system/yt-dlp-webui.service` yt-dlp webui can be launched as in background.
```
[Unit]
Description=yt-dlp-webui service file
After=network.target
[Service]
User=some_user
ExecStart=/usr/local/bin/yt-dlp-webui --out /mnt/share/downloads --port 8100
[Install]
WantedBy=multi-user.target
```
```shell
systemctl enable yt-dlp-webui
systemctl start yt-dlp-webui
```
It could be that yt-dlp-webui works correctly when started manually from the console, but with systemd, it does not see the yt-dlp executable, or has issues writing to the database file. One way to fix these issues could be as follows:
```shell
cd
mkdir yt-dlp-webui-workingdir
# optionally move the already existing database file there:
mv local.db yt-dlp-webui-workingdir
nano yt-dlp-webui-workingdir/my.conf
```
The config file format is described above; make sure to include the `downloaderPath` setting (the path can possibly be found by running `which yt-dlp`). For example, one could have:
```
downloadPath: /stuff/media
downloaderPath: /home/your_user/.local/bin/yt-dlp
log_path: /home/your_user/yt-dlp-webui-workingdir
session_file_path: /home/your_user/yt-dlp-webui-workingdir
```
Adjust the Service section in the `/etc/systemd/system/yt-dlp-webui.service` file as follows:
```
[Service]
User=your_user
Group=your_user
WorkingDirectory=/home/your_user/yt-dlp-webui-workingdir
ExecStart=/usr/local/bin/yt-dlp-webui --conf /home/your_user/yt-dlp-webui-workingdir/my.conf
```
## Manual installation
```sh
# the dependencies are: yt-dlp, ffmpeg, nodejs, go, make.
make all
```
## Open-API
Navigate to `/openapi` to see the related swagger.
## Extendable
You dont'like the Material feel?
Want to build your own frontend? We got you covered 🤠
`yt-dlp-webui` now exposes a nice **JSON-RPC 1.0** interface through Websockets and HTTP-POST
It is **planned** to also expose a **gRPC** server.
For more information open an issue on GitHub and I will provide more info ASAP.
## Custom frontend
To load a custom frontend you need to specify its path either in the config file ([see config file](#config-file)) or via flags.
The frontend needs to follow this structure:
```
path/to/my/frontend
├── assets
│ ├── js-chunk-1.js (example)
│ ├── js-chunk-2.js (example)
│ ├── style.css (example)
└── index.html
```
`assets` is where the resources will be loaded.
`index.html` is the entrypoint.
## Nix
This repo adds support for Nix(OS) in various ways through a `flake-parts` flake.
For more info, please refer to the [official documentation](https://nixos.org/learn/).
## What yt-dlp-webui is not
`yt-dlp-webui` isn't your ordinary website where to download stuff from the internet, so don't try asking for links of where this is hosted. It's a self hosted platform for a Linux NAS.
## Troubleshooting
- **It says that it isn't connected.**
- In some circumstances, you must set the server ip address or hostname in the settings section (gear icon).
- **The download doesn't start.**
- Simply, yt-dlp process takes a lot of time to fire up. (yt-dlp isn't fast especially if you have a lower-end/low-power NAS/server/desktop. Furthermore some yt-dlp builds are slower than others)
Bye :)

View File

@ -1,6 +1,6 @@
{
"name": "yt-dlp-webui",
"version": "3.2.5",
"version": "3.2.6",
"description": "Frontend compontent of yt-dlp-webui",
"scripts": {
"dev": "vite --host 0.0.0.0",
@ -18,11 +18,11 @@
"@mui/icons-material": "^6.2.0",
"@mui/material": "^6.2.0",
"fp-ts": "^2.16.5",
"jotai": "^2.10.3",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"react-router-dom": "^6.23.1",
"react-virtuoso": "^4.7.11",
"jotai": "^2.10.3",
"rxjs": "^7.8.1"
},
"devDependencies": {

View File

@ -28,6 +28,7 @@ import Footer from './components/Footer'
import Logout from './components/Logout'
import SocketSubscriber from './components/SocketSubscriber'
import ThemeToggler from './components/ThemeToggler'
import TwitchIcon from './components/TwitchIcon'
import { useI18n } from './hooks/useI18n'
import Toaster from './providers/ToasterProvider'
import { getAccentValue } from './utils'
@ -154,6 +155,19 @@ export default function Layout() {
<ListItemText primary={i18n.t('subscriptionsButtonLabel')} />
</ListItemButton>
</Link>
<Link to={'/twitch'} style={
{
textDecoration: 'none',
color: mode === 'dark' ? '#ffffff' : '#000000DE'
}
}>
<ListItemButton>
<ListItemIcon>
<TwitchIcon />
</ListItemIcon>
<ListItemText primary={"Twitch"} />
</ListItemButton>
</Link>
<Link to={'/monitor'} style={
{
textDecoration: 'none',

View File

@ -80,4 +80,7 @@ keys:
cronExpressionLabel: 'Cron expression'
editButtonLabel: 'Edit'
newSubscriptionButton: New subscription
clearCompletedButton: 'Clear completed'
clearCompletedButton: 'Clear completed'
twitchIntegrationInfo: |
To enable monitoring Twitch streams follow this wiki page.
https://github.com/marcopiovanello/yt-dlp-web-ui/wiki/Twitch-integration

View File

@ -121,14 +121,18 @@ export const appTitleState = atomWithStorage(
export const serverAddressAndPortState = atom((get) => {
if (get(servedFromReverseProxySubDirState)) {
return `${get(serverAddressState)}/${get(servedFromReverseProxySubDirState)}/`
.replaceAll('"', '') // TODO: atomWithStorage put extra double quotes on strings
.replaceAll('"', '') // XXX: atomWithStorage uses JSON.stringify to serialize
.replaceAll('//', '/') // which puts extra double quotes.
}
if (get(servedFromReverseProxyState)) {
return `${get(serverAddressState)}`
.replaceAll('"', '')
}
return `${get(serverAddressState)}:${get(serverPortState)}`
const sap = `${get(serverAddressState)}:${get(serverPortState)}`
.replaceAll('"', '')
return sap.endsWith('/') ? sap.slice(0, -1) : sap
})
export const serverURL = atom((get) =>
@ -137,12 +141,16 @@ export const serverURL = atom((get) =>
export const rpcWebSocketEndpoint = atom((get) => {
const proto = window.location.protocol === 'https:' ? 'wss:' : 'ws:'
return `${proto}//${get(serverAddressAndPortState)}/rpc/ws`
const sap = get(serverAddressAndPortState)
return `${proto}//${sap.endsWith('/') ? sap.slice(0, -1) : sap}/rpc/ws`
})
export const rpcHTTPEndpoint = atom((get) => {
const proto = window.location.protocol
return `${proto}//${get(serverAddressAndPortState)}/rpc/http`
const sap = get(serverAddressAndPortState)
return `${proto}//${sap.endsWith('/') ? sap.slice(0, -1) : sap}/rpc/http`
})
export const serverSideCookiesState = atom<Promise<string>>(async (get) => await pipe(

View File

@ -0,0 +1,22 @@
import { useAtomValue } from 'jotai'
import { settingsState } from '../atoms/settings'
const TwitchIcon: React.FC = () => {
const { theme } = useAtomValue(settingsState)
return (
<svg
role="img"
viewBox="0 0 24 24"
width={24}
height={24}
xmlns="http://www.w3.org/2000/svg"
style={{ fill: theme === 'dark' ? '#fff' : '#757575' }}
>
<title>Twitch</title>
<path d="M11.571 4.714h1.715v5.143H11.57zm4.715 0H18v5.143h-1.714zM6 0L1.714 4.286v15.428h5.143V24l4.286-4.286h3.428L22.286 12V0zm14.571 11.143l-3.428 3.428h-3.429l-3 3v-3H6.857V1.714h13.714Z" />
</svg>
)
}
export default TwitchIcon

View File

@ -0,0 +1,140 @@
import CloseIcon from '@mui/icons-material/Close'
import {
Alert,
AppBar,
Box,
Button,
Container,
Dialog,
Grid,
IconButton,
Paper,
Slide,
TextField,
Toolbar,
Typography
} from '@mui/material'
import { TransitionProps } from '@mui/material/transitions'
import { matchW } from 'fp-ts/lib/Either'
import { pipe } from 'fp-ts/lib/function'
import { useAtomValue } from 'jotai'
import { forwardRef, startTransition, useState } from 'react'
import { serverURL } from '../../atoms/settings'
import { useToast } from '../../hooks/toast'
import { useI18n } from '../../hooks/useI18n'
import { ffetch } from '../../lib/httpClient'
type Props = {
open: boolean
onClose: () => void
}
const Transition = forwardRef(function Transition(
props: TransitionProps & {
children: React.ReactElement
},
ref: React.Ref<unknown>,
) {
return <Slide direction="up" ref={ref} {...props} />
})
const TwitchDialog: React.FC<Props> = ({ open, onClose }) => {
const [channelURL, setChannelURL] = useState('')
const { i18n } = useI18n()
const { pushMessage } = useToast()
const baseURL = useAtomValue(serverURL)
const submit = async (channelURL: string) => {
const task = ffetch<void>(`${baseURL}/twitch/user`, {
method: 'POST',
body: JSON.stringify({
user: channelURL.split('/').at(-1)
})
})
const either = await task()
pipe(
either,
matchW(
(l) => pushMessage(l, 'error'),
(_) => onClose()
)
)
}
return (
<Dialog
fullScreen
open={open}
onClose={onClose}
TransitionComponent={Transition}
>
<AppBar sx={{ position: 'relative' }}>
<Toolbar>
<IconButton
edge="start"
color="inherit"
onClick={onClose}
aria-label="close"
>
<CloseIcon />
</IconButton>
<Typography sx={{ ml: 2, flex: 1 }} variant="h6" component="div">
{i18n.t('subscriptionsButtonLabel')}
</Typography>
</Toolbar>
</AppBar>
<Box sx={{
backgroundColor: (theme) => theme.palette.background.default,
minHeight: (theme) => `calc(99vh - ${theme.mixins.toolbar.minHeight}px)`
}}>
<Container sx={{ my: 4 }}>
<Grid container spacing={2}>
<Grid item xs={12}>
<Paper
elevation={4}
sx={{
p: 2,
display: 'flex',
flexDirection: 'column',
}}
>
<Grid container gap={1.5}>
<Grid item xs={12}>
<Alert severity="info">
{i18n.t('twitchIntegrationInfo')}
</Alert>
</Grid>
<Grid item xs={12} mt={1}>
<TextField
multiline
fullWidth
label={i18n.t('subscriptionsURLInput')}
variant="outlined"
placeholder="https://www.twitch.tv/a_twitch_user_that_exists"
onChange={(e) => setChannelURL(e.target.value)}
/>
</Grid>
<Grid item xs={12}>
<Button
sx={{ mt: 2 }}
variant="contained"
disabled={channelURL === ''}
onClick={() => startTransition(() => submit(channelURL))}
>
{i18n.t('startButton')}
</Button>
</Grid>
</Grid>
</Paper>
</Grid>
</Grid>
</Container>
</Box>
</Dialog>
)
}
export default TwitchDialog

View File

@ -1,6 +1,9 @@
import { tryCatch } from 'fp-ts/TaskEither'
import * as J from 'fp-ts/Json'
import * as E from 'fp-ts/Either'
import { pipe } from 'fp-ts/lib/function'
async function fetcher<T>(url: string, opt?: RequestInit): Promise<T> {
async function fetcher(url: string, opt?: RequestInit, controller?: AbortController): Promise<string> {
const jwt = localStorage.getItem('token')
if (opt && !opt.headers) {
@ -14,17 +17,27 @@ async function fetcher<T>(url: string, opt?: RequestInit): Promise<T> {
headers: {
...opt?.headers,
'X-Authentication': jwt ?? ''
}
},
signal: controller?.signal
})
if (!res.ok) {
throw await res.text()
}
return res.json() as T
return res.text()
}
export const ffetch = <T>(url: string, opt?: RequestInit) => tryCatch(
() => fetcher<T>(url, opt),
export const ffetch = <T>(url: string, opt?: RequestInit, controller?: AbortController) => tryCatch(
async () => pipe(
await fetcher(url, opt, controller),
J.parse,
E.match(
(l) => l as T,
(r) => r as T
)
),
(e) => `error while fetching: ${e}`
)

View File

@ -6,6 +6,7 @@ import Terminal from './views/Terminal'
const Home = lazy(() => import('./views/Home'))
const Login = lazy(() => import('./views/Login'))
const Twitch = lazy(() => import('./views/Twitch'))
const Archive = lazy(() => import('./views/Archive'))
const Settings = lazy(() => import('./views/Settings'))
const LiveStream = lazy(() => import('./views/Livestream'))
@ -111,6 +112,14 @@ export const router = createHashRouter([
</Suspense >
)
},
{
path: '/twitch',
element: (
<Suspense fallback={<CircularProgress />}>
<Twitch />
</Suspense >
)
},
]
},
])

View File

@ -0,0 +1,77 @@
import {
Chip,
Container,
Paper
} from '@mui/material'
import { matchW } from 'fp-ts/lib/Either'
import { pipe } from 'fp-ts/lib/function'
import { useAtomValue } from 'jotai'
import { useState, useTransition } from 'react'
import { serverURL } from '../atoms/settings'
import LoadingBackdrop from '../components/LoadingBackdrop'
import NoSubscriptions from '../components/subscriptions/NoSubscriptions'
import SubscriptionsSpeedDial from '../components/subscriptions/SubscriptionsSpeedDial'
import TwitchDialog from '../components/twitch/TwitchDialog'
import { useToast } from '../hooks/toast'
import useFetch from '../hooks/useFetch'
import { ffetch } from '../lib/httpClient'
const TwitchView: React.FC = () => {
const { pushMessage } = useToast()
const baseURL = useAtomValue(serverURL)
const [openDialog, setOpenDialog] = useState(false)
const { data: users, fetcher: refetch } = useFetch<Array<string>>('/twitch/users')
const [isPending, startTransition] = useTransition()
const deleteUser = async (user: string) => {
const task = ffetch<void>(`${baseURL}/twitch/user/${user}`, {
method: 'DELETE',
})
const either = await task()
pipe(
either,
matchW(
(l) => pushMessage(l, 'error'),
() => refetch()
)
)
}
return (
<>
<LoadingBackdrop isLoading={!users || isPending} />
<SubscriptionsSpeedDial onOpen={() => setOpenDialog(s => !s)} />
<TwitchDialog open={openDialog} onClose={() => {
setOpenDialog(s => !s)
refetch()
}} />
{
!users || users.length === 0 ?
<NoSubscriptions /> :
<Container maxWidth="xl" sx={{ mt: 4, mb: 8 }}>
<Paper sx={{
p: 2.5,
minHeight: '80vh',
}}>
{users.map(user => (
<Chip
label={user}
onDelete={() => startTransition(async () => await deleteUser(user))}
/>
))}
</Paper>
</Container>
}
</>
)
}
export default TwitchView

View File

@ -4,32 +4,39 @@ import (
"os"
"path/filepath"
"sync"
"time"
"gopkg.in/yaml.v3"
)
type Config struct {
LogPath string `yaml:"log_path"`
EnableFileLogging bool `yaml:"enable_file_logging"`
BaseURL string `yaml:"base_url"`
Host string `yaml:"host"`
Port int `yaml:"port"`
DownloadPath string `yaml:"downloadPath"`
DownloaderPath string `yaml:"downloaderPath"`
RequireAuth bool `yaml:"require_auth"`
Username string `yaml:"username"`
Password string `yaml:"password"`
QueueSize int `yaml:"queue_size"`
LocalDatabasePath string `yaml:"local_database_path"`
SessionFilePath string `yaml:"session_file_path"`
path string // private
UseOpenId bool `yaml:"use_openid"`
OpenIdProviderURL string `yaml:"openid_provider_url"`
OpenIdClientId string `yaml:"openid_client_id"`
OpenIdClientSecret string `yaml:"openid_client_secret"`
OpenIdRedirectURL string `yaml:"openid_redirect_url"`
FrontendPath string `yaml:"frontend_path"`
AutoArchive bool `yaml:"auto_archive"`
LogPath string `yaml:"log_path"`
EnableFileLogging bool `yaml:"enable_file_logging"`
BaseURL string `yaml:"base_url"`
Host string `yaml:"host"`
Port int `yaml:"port"`
DownloadPath string `yaml:"downloadPath"`
DownloaderPath string `yaml:"downloaderPath"`
RequireAuth bool `yaml:"require_auth"`
Username string `yaml:"username"`
Password string `yaml:"password"`
QueueSize int `yaml:"queue_size"`
LocalDatabasePath string `yaml:"local_database_path"`
SessionFilePath string `yaml:"session_file_path"`
path string // private
UseOpenId bool `yaml:"use_openid"`
OpenIdProviderURL string `yaml:"openid_provider_url"`
OpenIdClientId string `yaml:"openid_client_id"`
OpenIdClientSecret string `yaml:"openid_client_secret"`
OpenIdRedirectURL string `yaml:"openid_redirect_url"`
OpenIdEmailWhitelist []string `yaml:"openid_email_whitelist"`
FrontendPath string `yaml:"frontend_path"`
AutoArchive bool `yaml:"auto_archive"`
Twitch struct {
ClientId string `yaml:"client_id"`
ClientSecret string `yaml:"client_secret"`
CheckInterval time.Duration `yaml:"check_interval"`
} `yaml:"twitch"`
}
var (
@ -41,6 +48,7 @@ func Instance() *Config {
if instance == nil {
instanceOnce.Do(func() {
instance = &Config{}
instance.Twitch.CheckInterval = time.Minute * 5
})
}
return instance

View File

@ -100,6 +100,7 @@ func (p *Process) Start() {
templateReplacer.Replace(downloadTemplate),
"--progress-template",
templateReplacer.Replace(postprocessTemplate),
"--no-exec",
}
// if user asked to manually override the output path...

View File

@ -0,0 +1,21 @@
package middlewares
import (
"net/http"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/config"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/openid"
)
func ApplyAuthenticationByConfig(next http.Handler) http.Handler {
handler := next
if config.Instance().RequireAuth {
handler = Authenticated(handler)
}
if config.Instance().UseOpenId {
handler = openid.Middleware(handler)
}
return handler
}

View File

@ -6,10 +6,12 @@ import (
"encoding/json"
"errors"
"net/http"
"slices"
"time"
"github.com/coreos/go-oidc/v3/oidc"
"github.com/google/uuid"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/config"
"golang.org/x/oauth2"
)
@ -76,6 +78,21 @@ func doAuthentification(r *http.Request, setCookieCallback func(t *oauth2.Token)
return nil, err
}
var claims struct {
Email string `json:"email"`
Verified bool `json:"email_verified"`
}
if err := idToken.Claims(&claims); err != nil {
return nil, err
}
whitelist := config.Instance().OpenIdEmailWhitelist
if len(whitelist) > 0 && !slices.Contains(whitelist, claims.Email) {
return nil, errors.New("email address not found in ACL")
}
nonce, err := r.Cookie("nonce")
if err != nil {
return nil, err

View File

@ -164,7 +164,7 @@ func (s *Service) DeleteTemplate(ctx context.Context, id string) error {
func (s *Service) GetVersion(ctx context.Context) (string, string, error) {
//TODO: load from realease properties file, or anything else outside code
const CURRENT_RPC_VERSION = "3.2.5"
const CURRENT_RPC_VERSION = "3.2.6"
result := make(chan string, 1)

View File

@ -34,6 +34,7 @@ import (
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/status"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/subscription"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/subscription/task"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/twitch"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/user"
_ "modernc.org/sqlite"
@ -51,6 +52,7 @@ type serverConfig struct {
db *sql.DB
mq *internal.MessageQueue
lm *livestream.Monitor
tm *twitch.Monitor
}
// TODO: change scope
@ -115,17 +117,33 @@ func RunBlocking(rc *RunConfig) {
go lm.Schedule()
go lm.Restore()
srv := newServer(serverConfig{
tm := twitch.NewMonitor(
twitch.NewAuthenticationManager(
config.Instance().Twitch.ClientId,
config.Instance().Twitch.ClientSecret,
),
)
go tm.Monitor(
context.TODO(),
config.Instance().Twitch.CheckInterval,
twitch.DEFAULT_DOWNLOAD_HANDLER(mdb, mq),
)
go tm.Restore()
scfg := serverConfig{
frontend: rc.App,
swagger: rc.Swagger,
mdb: mdb,
mq: mq,
db: db,
lm: lm,
})
tm: tm,
}
go gracefulShutdown(srv, mdb)
go autoPersist(time.Minute*5, mdb, lm)
srv := newServer(scfg)
go gracefulShutdown(srv, &scfg)
go autoPersist(time.Minute*5, mdb, lm, tm)
var (
network = "tcp"
@ -188,12 +206,7 @@ func newServer(c serverConfig) *http.Server {
// Filebrowser routes
r.Route("/filebrowser", func(r chi.Router) {
if config.Instance().RequireAuth {
r.Use(middlewares.Authenticated)
}
if config.Instance().UseOpenId {
r.Use(openid.Middleware)
}
r.Use(middlewares.ApplyAuthenticationByConfig)
r.Post("/downloaded", filebrowser.ListDownloaded)
r.Post("/delete", filebrowser.DeleteFile)
r.Get("/d/{id}", filebrowser.DownloadFile)
@ -235,10 +248,18 @@ func newServer(c serverConfig) *http.Server {
// Subscriptions
r.Route("/subscriptions", subscription.Container(c.db, cronTaskRunner).ApplyRouter())
// Twitch
r.Route("/twitch", func(r chi.Router) {
r.Use(middlewares.ApplyAuthenticationByConfig)
r.Get("/users", twitch.GetMonitoredUsers(c.tm))
r.Post("/user", twitch.MonitorUserHandler(c.tm))
r.Delete("/user/{user}", twitch.DeleteUser(c.tm))
})
return &http.Server{Handler: r}
}
func gracefulShutdown(srv *http.Server, db *internal.MemoryDB) {
func gracefulShutdown(srv *http.Server, cfg *serverConfig) {
ctx, stop := signal.NotifyContext(context.Background(),
os.Interrupt,
syscall.SIGTERM,
@ -250,7 +271,9 @@ func gracefulShutdown(srv *http.Server, db *internal.MemoryDB) {
slog.Info("shutdown signal received")
defer func() {
db.Persist()
cfg.mdb.Persist()
cfg.lm.Persist()
cfg.tm.Persist()
stop()
srv.Shutdown(context.Background())
@ -258,8 +281,14 @@ func gracefulShutdown(srv *http.Server, db *internal.MemoryDB) {
}()
}
func autoPersist(d time.Duration, db *internal.MemoryDB, lm *livestream.Monitor) {
func autoPersist(
d time.Duration,
db *internal.MemoryDB,
lm *livestream.Monitor,
tm *twitch.Monitor,
) {
for {
time.Sleep(d)
if err := db.Persist(); err != nil {
slog.Warn("failed to persisted session", slog.Any("err", err))
}
@ -267,7 +296,10 @@ func autoPersist(d time.Duration, db *internal.MemoryDB, lm *livestream.Monitor)
slog.Warn(
"failed to persisted livestreams monitor session", slog.Any("err", err.Error()))
}
if err := tm.Persist(); err != nil {
slog.Warn(
"failed to persisted twitch monitor session", slog.Any("err", err.Error()))
}
slog.Debug("sucessfully persisted session")
time.Sleep(d)
}
}

75
server/twitch/auth.go Normal file
View File

@ -0,0 +1,75 @@
package twitch
import (
"encoding/json"
"fmt"
"net/http"
"net/url"
"time"
)
const authURL = "https://id.twitch.tv/oauth2/token"
type AuthResponse struct {
AccessToken string `json:"access_token"`
ExpiresIn int `json:"expires_in"`
TokenType string `json:"token_type"`
}
type AccessToken struct {
Token string
Expiry time.Time
}
type AuthenticationManager struct {
clientId string
clientSecret string
accesToken *AccessToken
}
func NewAuthenticationManager(clientId, clientSecret string) *AuthenticationManager {
return &AuthenticationManager{
clientId: clientId,
clientSecret: clientSecret,
accesToken: &AccessToken{},
}
}
func (a *AuthenticationManager) GetAccessToken() (*AccessToken, error) {
if a.accesToken != nil && a.accesToken.Token != "" && a.accesToken.Expiry.After(time.Now()) {
return a.accesToken, nil
}
data := url.Values{}
data.Set("client_id", a.clientId)
data.Set("client_secret", a.clientSecret)
data.Set("grant_type", "client_credentials")
resp, err := http.PostForm(authURL, data)
if err != nil {
return nil, fmt.Errorf("errore richiesta token: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("status non OK: %s", resp.Status)
}
var auth AuthResponse
if err := json.NewDecoder(resp.Body).Decode(&auth); err != nil {
return nil, fmt.Errorf("errore decoding JSON: %w", err)
}
token := &AccessToken{
Token: auth.AccessToken,
Expiry: time.Now().Add(time.Duration(auth.ExpiresIn) * time.Second),
}
a.accesToken = token
return token, nil
}
func (a *AuthenticationManager) GetClientId() string {
return a.clientId
}

91
server/twitch/client.go Normal file
View File

@ -0,0 +1,91 @@
package twitch
import (
"encoding/json"
"io"
"net/http"
"time"
)
const twitchAPIURL = "https://api.twitch.tv/helix"
type Client struct {
authenticationManager AuthenticationManager
}
func NewTwitchClient(am *AuthenticationManager) *Client {
return &Client{
authenticationManager: *am,
}
}
type streamResp struct {
Data []struct {
ID string `json:"id"`
UserName string `json:"user_name"`
Title string `json:"title"`
GameName string `json:"game_name"`
StartedAt string `json:"started_at"`
} `json:"data"`
}
func (c *Client) doRequest(endpoint string, params map[string]string) ([]byte, error) {
token, err := c.authenticationManager.GetAccessToken()
if err != nil {
return nil, err
}
reqURL := twitchAPIURL + endpoint
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
return nil, err
}
q := req.URL.Query()
for k, v := range params {
q.Set(k, v)
}
req.URL.RawQuery = q.Encode()
req.Header.Set("Client-Id", c.authenticationManager.GetClientId())
req.Header.Set("Authorization", "Bearer "+token.Token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return io.ReadAll(resp.Body)
}
func (c *Client) PollStream(channel string, liveChannel chan<- *StreamInfo) error {
body, err := c.doRequest("/streams", map[string]string{"user_login": channel})
if err != nil {
return err
}
var sr streamResp
if err := json.Unmarshal(body, &sr); err != nil {
return err
}
if len(sr.Data) == 0 {
liveChannel <- &StreamInfo{UserName: channel, IsLive: false}
return nil
}
s := sr.Data[0]
started, _ := time.Parse(time.RFC3339, s.StartedAt)
liveChannel <- &StreamInfo{
ID: s.ID,
UserName: s.UserName,
Title: s.Title,
GameName: s.GameName,
StartedAt: started,
IsLive: true,
}
return nil
}

149
server/twitch/monitor.go Normal file
View File

@ -0,0 +1,149 @@
package twitch
import (
"context"
"encoding/gob"
"fmt"
"iter"
"log/slog"
"maps"
"os"
"path/filepath"
"sync"
"time"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/config"
"github.com/marcopiovanello/yt-dlp-web-ui/v3/server/internal"
)
type Monitor struct {
liveChannel chan *StreamInfo
monitored map[string]*Client
lastState map[string]bool
mu sync.RWMutex
authenticationManager *AuthenticationManager
}
func NewMonitor(authenticationManager *AuthenticationManager) *Monitor {
return &Monitor{
liveChannel: make(chan *StreamInfo, 16),
monitored: make(map[string]*Client),
lastState: make(map[string]bool),
authenticationManager: authenticationManager,
}
}
func (m *Monitor) Add(user string) {
m.mu.Lock()
defer m.mu.Unlock()
m.monitored[user] = NewTwitchClient(m.authenticationManager)
slog.Info("added user to twitch monitor", slog.String("user", user))
}
func (m *Monitor) Monitor(ctx context.Context, interval time.Duration, handler func(url string) error) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
m.mu.RLock()
for user, client := range m.monitored {
u := user
c := client
go func() {
if err := c.PollStream(u, m.liveChannel); err != nil {
slog.Error("polling failed", slog.String("user", u), slog.Any("err", err))
}
}()
}
m.mu.RUnlock()
case stream := <-m.liveChannel:
wasLive := m.lastState[stream.UserName]
if stream.IsLive && !wasLive {
slog.Info("stream went live", slog.String("user", stream.UserName))
if err := handler(fmt.Sprintf("https://www.twitch.tv/%s", stream.UserName)); err != nil {
slog.Error("handler failed", slog.String("user", stream.UserName), slog.Any("err", err))
}
}
m.lastState[stream.UserName] = stream.IsLive
case <-ctx.Done():
slog.Info("stopping twitch monitor")
return
}
}
}
func (m *Monitor) GetMonitoredUsers() iter.Seq[string] {
m.mu.RLock()
defer m.mu.RUnlock()
return maps.Keys(m.monitored)
}
func (m *Monitor) DeleteUser(user string) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.monitored, user)
delete(m.lastState, user)
}
func DEFAULT_DOWNLOAD_HANDLER(db *internal.MemoryDB, mq *internal.MessageQueue) func(url string) error {
return func(url string) error {
p := &internal.Process{
Url: url,
Livestream: true,
Params: []string{"--downloader", "ffmpeg", "--no-part"},
}
db.Set(p)
mq.Publish(p)
return nil
}
}
func (m *Monitor) Persist() error {
filename := filepath.Join(config.Instance().SessionFilePath, "twitch-monitor.dat")
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
enc := gob.NewEncoder(f)
users := make([]string, 0, len(m.monitored))
for user := range m.monitored {
users = append(users, user)
}
return enc.Encode(users)
}
func (m *Monitor) Restore() error {
filename := filepath.Join(config.Instance().SessionFilePath, "twitch-monitor.dat")
f, err := os.Open(filename)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
defer f.Close()
dec := gob.NewDecoder(f)
var users []string
if err := dec.Decode(&users); err != nil {
return err
}
m.monitored = make(map[string]*Client)
for _, user := range users {
m.monitored[user] = NewTwitchClient(m.authenticationManager)
}
return nil
}

65
server/twitch/rest.go Normal file
View File

@ -0,0 +1,65 @@
package twitch
import (
"encoding/json"
"net/http"
"slices"
"github.com/go-chi/chi/v5"
)
type addUserReq struct {
User string `json:"user"`
}
func MonitorUserHandler(m *Monitor) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req addUserReq
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
m.Add(req.User)
if err := json.NewEncoder(w).Encode("ok"); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
}
func GetMonitoredUsers(m *Monitor) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
it := m.GetMonitoredUsers()
users := slices.Collect(it)
if users == nil {
users = make([]string, 0)
}
if err := json.NewEncoder(w).Encode(users); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
}
func DeleteUser(m *Monitor) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
user := chi.URLParam(r, "user")
if user == "" {
http.Error(w, "empty user", http.StatusBadRequest)
return
}
m.DeleteUser(user)
if err := json.NewEncoder(w).Encode("ok"); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
}

20
server/twitch/types.go Normal file
View File

@ -0,0 +1,20 @@
package twitch
import "time"
type StreamInfo struct {
ID string
UserName string
Title string
GameName string
StartedAt time.Time
IsLive bool
}
type VodInfo struct {
ID string
Title string
URL string
Duration string
CreatedAt time.Time
}