* Don't play video when seeking non-started video
* Set initial time on load instead of play
* Continue playing from current position when switching sources on error
* Remove unnecessary ref
* Don't log context canceled error during live transcode
* Pause live transcode if still scrubbing
* Debounce loading live transcode source to avoid multiple ffmpeg instances
* Don't start from start or resume time if seeking before playing
* Play video when seeked before playing
* Move optimise out of RunAllMigrations
* Separate read and write database connections
* Enforce readonly connection constraint
* Fix migrations not using tx
* #5155 - allow setting cache size from environment
* Document new environment variable
Per convo with people on Discord. I have updated the Reload Scrapers UI. It now adds a button if the filter box appears and then the button extends and takes up the whole space if the filter box does not exist.
---------
Co-authored-by: CJ <tedabed@gmail.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
I think was was happening is the browser was trying to do too much at once (Rendering the popup and focusing the input simultaneously). I believe it was triggering a reflow and setting the site back to default aka: back to the top.
I set the timeout to 0 which moves the execution to the next loop event. It gives the browser time to do one and then the other, not both at the same time.
I moved `onKeyPress` to `onKeyDown` due to the former being depreciated.
* Fix Tag and Alias odd spacing
As Echo6ix brough up the HTML Engine doesn't generate whitespace at the beginning of a string. Modifying it to use ` ` so that the spacing will be correct.
fixes https://github.com/stashapp/stash/issues/4997
* update for performerSelect and studioSelect
* Add default gallery image
* Add gallery cover URL path
* Use new cover URL in UI
* Hide gallery preview scrubber when gallery has no images
* Don't try to show lightbox for gallery without images
* Ignore unrelated lint issue
* Add UI support for setting containing groups
* Show containing groups in group details panel
* Move tag hierarchical filter code into separate type
* Add depth to scene_count and add sub_group_count
* Add sub-groups tab to groups page
* Add containing groups to edit groups dialog
* Show containing group description in sub-group view
* Show group scene number in group scenes view
* Add ability to drag move grid cards
* Add sub group order option
* Add reorder sub-groups interface
* Separate page size selector component
* Add interfaces to add and remove sub-groups to a group
* Separate MultiSet components
* Allow setting description while setting containing groups
* Show controls before video plays
* Allow interaction with controls while displaying error
* Source selector improvements
Don't auto-play next source if manually selected.
Don't remove errored sources
* Show errored sources in different style
* Support multiple calls to PluginApi.patch.instead for a component.
Allow calling the original/chained function from the hook function.
* Add example of new usage of instead
* Update documentation
* Fix filter reading from URL when not active
* Use alternative clone mechanism. Fixes weird filter hook behaviour
* Separate search term input component
* Refactor list filter to use contexts
* Refactor FilteredListToolbar
* Move components into separate files
* Convert ItemList hook into components
* Fix criteria clone functions
* Add toggle for sub-studio content
* Add toggle for sub-tag content
* Make LoadingIndicator height smaller and fade in.
* Copy apikey query parameter to DASH & HLS manifest
When an API key is provided to the DASH and HLS manifest endpoints, this
it will now be copied to the URLs inside the manifest. This allows for
clients that are only able to pass an URL to an (external) video player
to function in case authentication is set up on stash.
* Refactor repeated code into BackgroundImage
* Move BackgroundImage into Details folder
* Refactor performer tabs
* Refactor studio tabs
* Refactor tag tabs
* Refactor repeated code into DetailTitle
* Refactor repeated collapse button code into component
* Reuse FavoriteIcon in details pages
* Refactor performer urls into component
* Refactor alias list into component
* Refactor repeated image code into HeaderImage and LightboxLink components
* Replace render functions with inline conditional rendering
* Support new twitter hostname
* Rename Movie and MoviePartial to Group/GroupPartial
* Rename Movie interfaces
* Update movie url builders to use group
* Rename movieRoutes to groupRoutes
* Update dataloader
* Update names in sqlite package
* Rename in resolvers
* Add GroupByURL to scraper config
* Scraper backward compatibility hacks
* Refactor scraping settings panel
* Add max-height to scraper table
* Separate scraper section
* Add filter to scrapers section
* Add counters to scraper headings
* Show all urls with a scrollbar
* Sort URLs
* Move scene scraper menu into reusable component
* Reuse ScraperMenu for scene query menu
* Reuse scraper menu in GalleryEditPanel
* Add filter to scraper menu
* Add divider between stashboxes and scrapers
* Replace movies with groups in the UI
* Massage menu items
* Change view names
* Rename Movie components to Group
* Refactor movie to group variable names
* Rename movie class names to group
I think not including the scene_id meant that a date could be corrected earlier, meaning the rows affected would be 0. Adding scene_id means that each row should be migrated one by one.
* Fixes format in full hw encoding to nv12 for cuda, vaapi and qsv now
* Remove extra_hw_frames
* Add apple transcoder support
* Up the duration to discover decoding errors
* yuv420p is not supported on intel
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix makeTagFilter mode
* Remove studio_tags filter criterion
This is handled by studios_filter. The support for this still needs to be added in the UI, so I have removed the criterion options in the short-term.
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make pagination more compact
Support entering page number or clicking from drop down
* Fix border radius in dropdown in btn group
* Separate page count control
* Move filter criterion handlers into separate file
* Add related filters for image filter
* Add related filters for scene filter
* Add related filters to gallery filter
* Add related filters to movie filter
* Add related filters to performer filter
* Add related filters to studio filter
* Add related filters to tag filter
* Add scene filter to scene marker filter
* Update to Go 1.22
Updates to Go 1.22 because 1.19 is un-supported and has some CVEs.
Also updates a small number of low-risk deps
* Explicitly install Go in CI
* Bump compiler version
* Add build tags to it target
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix return types for RegisterComponent and PatchFunction
* Add support for patching TagSelect.sort
* Add support for patching PerformerSelect.sort
* Patch other select component sort functions
* Document patchable functions/components
* Add scene detail header
* Make common count button and add view count
* Add titles to play count and o count buttons
* Move rating from edit panel
* Include frame rate in header
* Remove redundant title/studio
* Improve numeric rating presentation
* Add star where there is no rating header
* Set rating on blur when click to edit
* Add star to numeric rating on gallery wall card
* Apply click to rate on movie page
* Apply click to rate to performer page
* Apply click to rate to studio page
* Fix rating number presentation on list tables
* Add data-value attributes
* Make ffmpeg/ffprobe settable and remove auto download
* Detect when ffmpeg not present in setup
* Add download ffmpeg task
* Add download ffmpeg button in system settings
* Download ffmpeg during setup
* Sort Performers by Last O / View
Added 2 New Sorts 'Last O At' and 'Last Played At' for Performers
* Filter Performers by Play Count
Was not sure whether to label this 'views' as the code does, or 'plays' but chose the latter as it gives parity across the scenes and performers filters.
* Sort Performers by Play Count
Reutilised the prior selectPerformerLastOAtSQL code that was used to filter by play count to additionally provide useful sorting options.
* Replaced O-Counter with O Count
To better match other sort and filter options like Gallery Count, Image Count, Play Count, Scene Count, Tag Count, File Count, Performer Count and Play Count, we should really use O Count rather than O-Counter for increased legibility and coherence.
* Title Case on 'Interactive speed' and correct capitalization for 'phash'
Every other filter/sort option is using Title Case other than 'Interactive speed' which stands out as incorrect. Also, fixing the correct mid-word capitalization on phash to pHash.
* Formatting
Formatted source code and Ran all tests
* Support setting nested UI values
* Accept partial for configureUI
* Send partial UI
* Save scan, generate and auto-tag options on change
* Send partials in saveUI
* Save library task options on change
* Update StashDB details in README.md
- Directs users to new guide in the StashDB docs instead of Discord
- No longer necessary to join Discord/Matrix for new users of StashDB now that invite codes are multi-use
- Updates formatting of the same "Quickstart Guide" section a little
* Expands quickstart language based on DogmaDragon's suggestions
* Accept plain map for runPluginTask
* Support running plugin task without task name
* Add interface to run plugin operations
* Update RunPluginTask client mutation
* Add ids to findMovies input
* Use ids for other find interfaces
* Update client side
* Fix gallery select function
* Replace movie select
* Re-add creatable
* Overhaul movie table
* Remove and deprecated unused code
* Change default toast placement
* Position at bottom on mobile
* Show single toast message at a time
* Optionally show dialog for error messages
* Fix circular dependency
* Animate toast
* Add interface to load tags by id
* Use minimal data for tag select queries
* Center image/text in select list
* Overhaul tag select
* Support excludeIds. Comment out image in dropdown
* Replace existing selects
* Remove unused code
* Fix styling of aliases
* Add Plugins Path setting
* Fix/improve cache invalidation
* Hide load error when collapsing package source
* Package manager style tweaks
* Show error if installed packages query failed
* Prevent "No packages found" flicker
* Show <unknown> if empty version
* Always show latest version, highlight if new version available
* Fix issues with non-unique cross-source package ids
* Don't wrap id, version and date
* Decrease collapse button padding
* Display description for scraper packages
* Fix default packages population
* Change default package path to community
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Set PYTHONPATH environment variable for Python script scrapers
* Convert PYTHONPATH to absolute
* Generalise and apply to plugins
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Update vrmode.ts
Enabling the most common VR projection (180_LR) and an older but no longer used one (360_TB) in the UI.
* Downgrade videojs-vr
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Added Studio Code and Photographer to Galleries
* Fix gallery display on mobile
* Fixed potential panic when scraping with a bad configuration
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Log more when resolving Python
Users often have problems configuring their Python installations
* Convert if-else ladder to switch statement
* Consolidate Python resolution
Adds additional logging to plugin tasks to
align with the logging that scrapers output.
* Add assets for plugins
* Move plugin javascript and css into separate endpoints
* Allow loading external scripts
* Add csp overrides
* Only include enabled plugins
* Move URLMap to utils
* Use URLMap for assets
* Add documentation
* Fix macOS notifications
* Change CFBundleIdentifier to match domain
* Distribute Stash.app
* Also build universal phasher binary
* Fix binary name in check_version.go
* Expose GOOS, working dir and home dir in systemStatus endpoint
* Disable setup in working directory when running Stash.app
* More Makefile improvements, remove unused scripts
* Improve READMEs and documentation
* Update a number of dependencies (incl. CVE fixes)
Includes some dependencies that were upgraded in #4106 as well as a few more dependencies.
Some deps that have been upgraded had CVEs.
Notably, upgrades deprecated dependencies such as:
- `github.com/go-chi/chi` (replaced with `/v5`)
- `github.com/gofrs/uuid` (replaced with `/v5`)
- `github.com/hashicorp/golang-lru` (replaced with `/v2` which uses generics)
* Upgraded a few more deps
* lint
* reverted yaml library to v2
* remove unnecessary mod replace
* Update chromedp
Fixes#3733
* Update Lightbox.tsx to also change default delay here to 5 sec instead of 5000
* Update config.go to set default slideshow delay from 5000 sec to 5 sec
Testing indicated that the arm-linux-gnueabihf-gcc compiler was emitting a binary that used thumb2 directives that are illegal on raspberry pi. Using the arm-linux-gnueabi-gcc compiler appears to fix the issue. Added march directive to target v7.
* Fix false positive mismatch in Movie Scrape dialog
Scraping a movie by URL would show a difference in duration because the
persisted value of duration was converted to HH:MM:SS while the newly
scraped value was displayed without formatting: this makes sense because
the newly scraped value is just a string and so could be anything
This adds a check to see if the string is a number and converts it to
HH:MM:SS format if possible
* Fallback to original value if not a number
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Exclude value for is null/not null
Also includes changes to the error message in the migration to include the filter string.
* Ignore null when setting from encoded criterion
* UI Update to show which file is being deleted
Added Selection dropwdown
Added checkbox to ensure that the codecs are the same within the group
* Refactor size options
* Convert select box to dropdown
* Internationalisation
---------
Co-authored-by: Steve Enderby <vpn-enderbys@capitatflpp.onmicrosoft.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add mockery config file
* Move basic file/folder structs to models
* Fix hack due to import loop
* Move file interfaces to models
* Move folder interfaces to models
* Move scene interfaces to models
* Move scene marker interfaces to models
* Move image interfaces to models
* Move gallery interfaces to models
* Move gallery chapter interfaces to models
* Move studio interfaces to models
* Move movie interfaces to models
* Move performer interfaces to models
* Move tag interfaces to models
* Move autotag interfaces to models
* Regenerate mocks
* yarn add videojs-abloop
* add abLoop plugin to video player
* adding player keyboard shortcut 'l' for toggling a/b looping
copies mpv behavior:
if a/b loop start not yet set, sets start to current player time
elif a/b loop stop not yet set, sets end to current player time and enables loop
else, disables a/b loop
relates to #3264 (https://github.com/stashapp/stash/issues/3264)
* update help with keyboard shortcut
* Add plugin type definitions
* Make UI elements optional
---------
Co-authored-by: chickenwingavalanche <chickenwingavalanche@example.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Move loadStickyHeader to src/hooks
* intl stashIDs
* Scroll to top on component mount
* Add id to gallery cover image and tweak merge functions
* Add useTitleProps hook
* Also scroll to top on list pages
* Refactor loaders and tabs
* Use classnames
* Add DetailImage
* mobile improvements to performer page
* updated remaining details pages
* fixes tag page on mobile
* implemented show hide for performer details
* fixes card width cutoff on mobile(not related to redesign)
* added background image option plus more improvements
* add tooltip for age field
* translate encoding message string
* Studio image and parent studio support in scene tagger
* Refactor studio backend and add studio tagger
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix scene marker NOT NULL constraint error
* similar changes to gallery chapters
* Fix NULL conversion error if names are NULL in DB
* Fix scene marker form resetting
* Add URLs scene relationship
* Update unit tests
* Update scene edit and details pages
* Update scrapers to use urls
* Post-process scenes during query scrape
* Update UI for URLs
* Change urls label
* Add `-v/--version` flag to print version string
- Created a new flag `-v/--version` in the command-line interface to display the version number and exit.
- Moved all version-related functions inside the config package to the new file `manager/config/version.go` to avoid circular dependencies.
- Added a new `GetVersionString()` function to generate a formatted version string.
- Updated references to the moved version functions.
- Updated references in the `Makefile`.
* Move version embeds to build package
* Remove githash var
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add phasher
A simple `phasher` program that accepts a video file as a command line
argument and calculates and prints its PHASH.
The goal of this separate executable is to have a simple way to
calculate phashes that doesn't depend on a full stash instance so that
third-party systems and tools can independently generate PHASHes which
can be used for interacting with stash and stash-box APIs and data.
Currently `phasher` is built in the default make target along with
`stash` by simply running `make`.
Cross-platform targets have not been considered.
Concurrency is intentionally not implemented because it is simpler to
use [GNU Parallel](https://www.gnu.org/software/parallel/).
For example:
```
parallel phasher {} ::: *.mp4
```
* standard dir structure for phasher and separate make target
The make target still needs to be integrated into the rest of the
Makefile so it can be built as part of normal releases.
* phasher: basic usage output and quiet option
* phasher: allow and process multiple command line arguments
* phasher: camelCase identifiers
* phasher: initialize ffmpeg and ffprobe only once
* Update text.ts
Displayed resolutions in Stash were confusing as hell when it came to VR files - which are typically 2:1. Now I understand why, it's assuming 16:9 files/looking at height only.
* Remove ID from PerformerPartial
* Separate studio model from sqlite model
* Separate movie model from sqlite model
* Separate tag model from sqlite model
* Separate saved filter model from sqlite model
* Separate scene marker model from sqlite model
* Separate gallery chapter model from sqlite model
* Move ErrNoRows checks into sqlite, improve empty result error messages
* Move SQLiteDate and SQLiteTimestamp to sqlite
* Use changesetTranslator everywhere, refactor for consistency
* Make PerformerStore.DestroyImage private
* Fix rating on movie create
* Use debounce hook
* Wait until search request complete before refreshing results
* Add back null modifiers
* Convert old excludes criterion to includes criterion
* Display criteria with only excludes items as excludes
* Fix depth display
* Reset search after selection
* Add back is modifier to tag filter
* Focus the input dialog after select/unselect
* Update unsupported modifiers
* Fix joined hierarchical filtering
* Fix scene performer tag filter
* Generalise performer tag handler
* Add unit tests
* Add equals handling
* Make performer tags equals/not equals unsupported
* Make tags not equals unsupported
* Make not equals unsupported for performers criterion
* Support equals/not equals for studio criterion
* Fix marker scene tags equals filter
* Fix scene performer tag filter
* Make equals/not equals unsupported for hierarchical criterion
* Use existing studio handler in movie
* Hide unsupported tag modifier options
* Use existing performer tags logic where possible
* Restore old parent/child filter logic
* Disable sub-tags in equals modifier for tags criterion
* Fix the fix for displayed performer image sticking after save
* Refactor for consistency
* Fully extract entity create/update logic from edit pages
* Fix submit hotkeys
* Refactor scene cover preview
* Fix atoi error on new scene page
* Support excludes field
* Refactor studio filter
* Refactor tags filter
* Support excludes in tags
---------
Co-authored-by: Kermie <kermie@isinthe.house>
* Add penis length stat to performers.
* Modified the UI to display and edit the stat.
* Added the ability to filter floats to allow filtering by penis length.
* Add circumcision stat to performer.
* Refactor enum filtering
* Change boolean filter to radio buttons
* Return null for empty enum values
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Use pills for modifier selector
* Fix caption default modifier
* Increase clickable area for criterion remove
If the area becomes too large, we can use half margin half padding.
Reduces the amount of pixel hunting required to click.
* Use pill-styled buttons
* GalleryInExClusion // Create Gallery from folder based on file, short description in setting
* GalleryInExClusion // No Folderiteration, expansion of docs
* GalleryInExClusion // Only accept lowercase files
* GalleryInExClusion // Correct text in settings
* Close input file so SafeMove can delete it
This is happening on Windows and over the network but at the end of SafeMove it fails the move with an error that it can't remove the input because it is in use.
It turns out it is in use by the SafeMove itself :)
* Copy the src file mod time
- (As far as I know,) scraping is irrelevant to auto tagging so I
removed it from the Auto Tagging documentation. Alternatively, it
could be moved to the bottom.
Co-authored-by: Bawdy Ink Slinger <BawdyInkSlinger@gmail.com>
`stash --help` exits with a non-zero exit code. Because `stash --help`
is a legitimate invocation, it should return an exit code of zero.
Adding an explicit help flag allows for exiting with a successful exit
code.
* Pinned Filters // Add the ability to pin filters in the new filter dialog
* Pinned Filters // Prevent overlap with x
* Pinned Filters // Pills in the button show up correctly now...
* Pinned Filters // Maximum height for mobile view
* Pinned Filters // Save in config.yml
* Style changes and minor fixes
* Pinned Filters // Increase divider space
---------
Co-authored-by: DingDongSoLong4 <99329275+DingDongSoLong4@users.noreply.github.com>
* initial commit of sort performer by o-count
* work on o_counter filter
* filter working
* sorting, filtering using combined scene+image count
* linting
* fix performer list view
---------
Co-authored-by: jpnsfw <none@none.com>
On a new installation, Stash will default to 'blobs' (which relative to the CWD of /, becomes '/blobs').
In our recommended installation, this is NOT mapped and as a result will NOT be persistent across the container being removed or updated.
So by default, we are populating blob data which will be deleted, likely leaving to some confused new users.
This should also be noted for existing installations to make this a persistent path.
* Fix error if movie back image blob was not found
* Don't error out if scene cover get fails
* Don't error out on image get fails
* Add debug logging for fs blobs
* Remove old blob data when no longer referenced
Current docker variable to pass localtime to container does not function (`/etc/localtime` does not exist within a running container). I've never built a container, nor personally use Alpine, but based on my digging it appears the package `tzdata` is required for this functionality
* Add date picker dependency
* Add DateInput component
* Add DateInput to edit panels
* Add DateInput to DateFilter
* Add time to DateInput and add to Timestamp filter
* Use calendar icon for button
* Refactor transaction hooks. Add preCommit
* Add BlobStore
* Use blobStore for tag images
* Use blobStore for studio images
* Use blobStore for performer images
* Use blobStore for scene covers
* Don't generate screenshots in legacy directory
* Run post-hooks outside original transaction
* Use blobStore for movie images
* Remove unnecessary DestroyImage methods
* Add missing filter for scene cover
* Add covers to generate options
* Add generate cover option to UI
* Add screenshot migration
* Delete thumb files as part of screenshot migration
* HW Accel
* CUDA Docker build and adjust the NVENC encoder
* Removed NVENC preset
Using legacy presets is removed in SDK 12 and deprecated since SDK 10.
This commit removed the preset to allow ffmpeg to select the default one.
---------
Co-authored-by: Nodude <>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Added the ability to do Seqential Scans
* Modify pkg/txn to run hooks with the outer context, instead of the context that was in a transaction
* update in application manual
* Add range to funscript heatmaps
* Add config to draw heatmap range
* Add setting to UI
* Apply draw range setting
Includes some refactoring
---------
Co-authored-by: kermieisinthehouse <kermie@isinthe.house>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make the indicator transparent and add a background-size class
* Show the heatmap if available
* Styling tweaks
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add shortcuts for decimal rating
* Add shortcut to reset decimal rating
* Generalise rating keybind code
Use r x x for decimal ratings.
* Update manual page
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix possible infinite loop/stack overflow with weird/broken zip files
* Fix path length calculation using bytes instead of characters (runes)
* Fix bug where oshash gets buffers with size not actually multiple of 8
* Add oshash tests
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add counter to File Info where file count > 1
* Add file modification time to File Info panel
* Remove duplicate intl keys
* Add file count to duplicate checker
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Treat no output from ffmpeg as an error condition
* Distinguish file vs. video duration, and use later where appropriate
* Check for empty file in generateFile
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* track watchtime and view time
* add view count sorting, added continue position filter
* display metrics in file info
* add toggle for tracking activity
* save activity every 10 seconds
* reset resume when video is nearly complete
* start from beginning when playing scene in queue
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make read-only operations use WithReadTxn
* Allow one database write thread
* Add unit test for concurrent transactions
* Perform some actions after commit to release txn
* Suppress some errors from cancelled context
* graphql: support date and timestamp filter types
* sql: add support for date & timestamp criterions
* ui: add support for date and timestamp criterions
* scenes: add support for filtering by date, created at and updated at
* image: support filtering by created at and updated at
* gallery: support filtering by date, created at and updated at
* movie: support filtering by date, created at and updated at
* studio: support filtering by date, created at and updated at
* tag: support filtering by date, created at and updated at
* performer: support filtering by bitrh & death date and created & updated at
* marker: support filtering by created & updated at and scene date, created & updated at
* Reassign scene file functionality
* Implement scene create
* Add scene create UI
* Add sceneMerge backend support
* Add merge scene to UI
* Populate split create with scene details
* Add merge button to duplicate checker
* Handle file-less scenes in marker preview generate
* Make unique file name for file-less scene exports
* Add o-counter to scene update input
* Hide rescan for file-less scenes
* Generate heatmap if no speed set on file
* Fix count in scene/image queries
Introducing a limit to how many options are shown in select dropdowns. Fixes an issue I was experiencing where large numbers of options (5000 tags) was causing dropdown to be unresponsive. Does not effect filtering, always shows 'Create "..."' option if it exists, and shows a notice at the bottom of the dropdown of how many options were hidden from the list if any were.
* added schema migration and updated data models
* added code and director to UI
* new fields are exported and imported
* added filters
* Add changelog entry
* Add types to player plugins
* Use videojs-vtt.js to parse sprite VTT files
* Overhaul scene player
* Replace vtt-thumbnails-freetube
* Remove chapters_vtt
* Force remove shadow from player progress bar
* Cleanup player css
* Rewrite live.ts as middleware
* Don't force play when changing source
* Fire handlers when file updated or moved
* Create galleries as needed
* Clean empty galleries
* Handle cleaning zip folders when path changed
* Fix gallery association on duplicate images
* Re-create missing folder-based galleries
* Only update fingerprints if changed
* Fix panic when loading primary file fails
* Fix gallery/scene association
* Fix display of scene gallery in card
* Use natural_cs collation with paths for title sorting
* Don't recalculate MD5 if not enabled
Remove MD5 if oshash has changed and MD5 was not calculated.
* Fix panic in paged DLNA
* Prevent identical hashes in stash-box drafts
* Fix incorrect timestamp updates
* Correct folder time fields
* Add migration with new indexes
* Correct mod_time format
* Add mod_time to data massage
* Load scene relationships on demand
* Load image relationships on demand
* Load gallery relationships on demand
* Add dataloaden
* Use dataloaders
* Use where in for other find many functions
* Do database txn in same thread. Retry on locked db
* Remove captions from slimscenedata
* Fix tracing
* Use where in instead of individual selects
* Remove scenes_query view
* Remove image query view
* Remove gallery query view
* Use where in for FindMany
* Don't interrupt scanning zip files
* Fix image filesize sort
* Use cache during migration
* Avoid use of query views
* Use FindMany to find related objects
* Log slow queries
* Add folders to generated files
* Use SlimScene for scene queries
* Include filename in migration error message
* Fix destroy gallery not destroying file
* Re-add minModTime functionality
* Deprecate useFileMetadata and stripFileExtension
* Optimise files post migration
* Decorate moved files. Use first missing file in move
* Include path in thumbnail generation error log
* Fix stash-box draft submission
* Don't destroy files unless deleting
* Call handler for files with no associated objects
* Fix moved zips causing error on scan
* Restructure data layer part 2 (#2599)
* Refactor and separate image model
* Refactor image query builder
* Handle relationships in image query builder
* Remove relationship management methods
* Refactor gallery model/query builder
* Add scenes to gallery model
* Convert scene model
* Refactor scene models
* Remove unused methods
* Add unit tests for gallery
* Add image tests
* Add scene tests
* Convert unnecessary scene value pointers to values
* Convert unnecessary pointer values to values
* Refactor scene partial
* Add scene partial tests
* Refactor ImagePartial
* Add image partial tests
* Refactor gallery partial update
* Add partial gallery update tests
* Use zero/null package for null values
* Add files and scan system
* Add sqlite implementation for files/folders
* Add unit tests for files/folders
* Image refactors
* Update image data layer
* Refactor gallery model and creation
* Refactor scene model
* Refactor scenes
* Don't set title from filename
* Allow galleries to freely add/remove images
* Add multiple scene file support to graphql and UI
* Add multiple file support for images in graphql/UI
* Add multiple file for galleries in graphql/UI
* Remove use of some deprecated fields
* Remove scene path usage
* Remove gallery path usage
* Remove path from image
* Move funscript to video file
* Refactor caption detection
* Migrate existing data
* Add post commit/rollback hook system
* Lint. Comment out import/export tests
* Add WithDatabase read only wrapper
* Prepend tasks to list
* Add 32 pre-migration
* Add warnings in release and migration notes
* Use primitive string in recommendation row props
* Use unique keys in recommendation rows
The keys for the cards used while loading clash with the ids of the actual cards, causing a list unique key warning.
* List filter alignment tweaks
* Rework list hook filtering
* Internationalise checksum correctly
* Use hotkeys '[' and ']' to scrub video player forwards and backwards by 10% of the scene
* Don't loop back to beginning
* Add manual keybind entry
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* refactored common code in recommendation row
* Implement front page options in config
* Allow customisation from front page
* Rename recommendations to front page
* Add generic UI settings
* Support adding premade filters
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
The incorrect error check means that images is always 0, which means scanImages is also always true, which causes the images inside the zip to be unnecessarily rescanned every time.
* Changed Most Active Studios to Latest Studios
* dynamically create view all link and created message for view all
* created shared determineSlidesToScroll method
* removed added code in Shared/index.ts
* renamed getSlickSettings to getSlickSliderSettings
* Updated row headers to follow plex naming convention
* removed extra s in Sceness
* updated row header css to better align header text with view all anchor
* Refactor interactive into context
* Stop the interactive device when leaving page
* Show interactive state if not ready
* Handle navigation and looping
* Rename manager.instance to Manager
* Show dialog message on fatal error on Windows
* Hide console windows explicitly on icon launch
Gets rid of the windowsgui flag, which causes all sorts of issues.
Instead, checks if stash was launched from the icon, and if so hides the console.
* Remove fixconsole
* Add changelog entries
* Make the script scraper context-aware
Connect the context to the command execution. This means command
execution can be aborted if the context is canceled. The context is
usually bound to user-interaction, i.e., a scraper operation issued
by the user. Hence, it seems correct to abort a command if the user
aborts.
* Enable errchkjson
Some json marshal calls are *safe* in that they can never fail. This is
conditional on the types of the the data being encoded. errchkjson finds
those calls which are unsafe, and also not checked for errors.
Add logging warnings to the place where unsafe encodings might happen.
This can help uncover usage bugs early in stash if they are tripped,
making debugging easier.
While here, keep the checker enabled in the linter to capture future
uses of json marshalling.
* Pass the context for zip file scanning.
* Pass the context in scanning
* Pass context, replace context.TODO()
Where applicable, pass the context down toward the lower functions in
the call stack. Replace uses of context.TODO() with the passed context.
This makes the code more context-aware, and you can rely on aborting
contexts to clean up subsystems to a far greater extent now.
I've left the cases where there is a context in a struct. My gut feeling
is that they have solutions that are nice, but they require more deep
thinking to unveil how to handle it.
* Remove context from task-structs
As a rule, contexts are better passed explicitly to functions than they
are passed implicitly via structs. In the case of tasks, we already
have a valid context in scope when creating the struct, so remove ctx
from the struct and use the scoped context instead.
With this change it is clear that the scanning functions are under a
context, and the task-starting caller has jurisdiction over the context
and its lifetime. A reader of the code don't have to figure out where
the context are coming from anymore.
While here, connect context.TODO() to the newly scoped context in most
of the scan code.
* Remove context from autotag struct too
* Make more context-passing explicit
In all of these cases, there is an applicable context which is close
in the call-tree. Hook up to this context.
* Simplify context passing in manager
The managers context handling generally wants to use an outer context
if applicable. However, the code doesn't pass it explicitly, but stores
it in a struct. Pull out the context from the struct and use it to
explicitly pass it.
At a later point in time, we probably want to handle this by handing
over the job to a different (program-lifetime) context for background
jobs, but this will do for a start.
* Upgrade gqlgen to v0.17.2
This enables builds on Go 1.18. github.com/vektah/gqlparser is upgraded
to the newest version too.
Getting this to work is a bit of a hazzle. I had to first remove
vendoring from the repository, perform the upgrade and then re-introduce
the vendor directory. I think gqlgens analysis went wrong for some
reason on the upgrade. It would seem a clean-room installation fixed it.
* Bump project to 1.18
* Update all packages, address gqlgenc breaking changes
* Let `go mod tidy` handle the go.mod file
* Upgrade linter to 1.45.2
* Introduce v1.45.2 of the linter
The linter now correctly warns on `strings.Title` because it isn't
unicode-aware. Fix this by using the suggested fix from x/text/cases
to produce unicode-aware strings.
The mapping isn't entirely 1-1 as this new approach has a larger iface:
it spans all of unicode rather than just ASCII. It coincides for ASCII
however, so things should be largely the same.
* Ready ourselves for errchkjson and contextcheck.
* Revert dockerfile golang version changes for now
Co-authored-by: Kermie <kermie@isinthe.house>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Replace JW Player with video.js
* Move HLS stream to bottom of list
HLS doesn't work very well on non-ios devices.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Move main to cmd
* Move api to internal
* Move logger and manager to internal
* Move shell hiding code to separate package
* Decouple job from desktop and utils
* Decouple session from config
* Move static into internal
* Decouple config from dlna
* Move desktop to internal
* Move dlna to internal
* Decouple remaining packages from config
* Move config into internal
* Move jsonschema and paths to models
* Make ffmpeg functions private
* Move file utility methods into fsutil package
* Move symwalk into fsutil
* Move single-use util functions into client package
* Move slice functions to separate packages
* Add env var to suppress windowsgui arg
* Move hash functions into separate package
* Move identify to internal
* Move autotag to internal
* Touch UI when generating backend
* Continue identify if source fails
* Handle empty result set correctly
* Parse null values from scraper script correctly
* Omit warning when json selector value missing
* Return nil when scraped item not found
* Fix graphql validation errors
* Add duration to autotag finish message
* No sorting scene/image/gallery where not specified
* Use an LRU cache for sqlite regexp function
* Compile path separator regex once
* Cache objects with single letter first names
* Move finished auto-tag log
* Add more verbose logging
* Add new changelog
* Remove single unicode character from autotag query
* Compile regex once where possible
* Fix CPU profiling
* Only match unicode characters if in path
* Performer tagger modal will load country name instead of code
When using the performer tagger, it would pull the 2 character country code from StashDB and store that. It now shows the full country name and stores the full country name instead. Also verified that it causes no issue with a performer that has no country set on StashDB
* Make changes to fix special characters in Criterion labels (#1819)
Reverse the '&' and '+' replacement done on StringCriterion
Decodes special characters in IHierarchicalLabeledIdCriterion
* Have Gallery info panel say "Downloaded From" next to URL
In the gallery info panel it says "Path" next to the URL. This updates it to say "Downloaded From" next to the URL, like it does in the scene view for the URL.
* Add country to EditPerformersDialog
* Add most text fields to EditPerformersDialog
* Refactor to pass validate
* Remove height and measurements fields
* Add gender field
* Run fmt-ui
* Cleaned up some language.
- Optimized the Quick Start guide, moved "how do i get to stash" from FAQ, moved some sections together.
* as above
* Update README.md
Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
* Some editing and notes
* Remove double click line, let's not patronize :)
Co-authored-by: kermieisinthehouse <kermie@isinthe.house>
Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
* Exposed created_at and updated_at dates on the detail panels for images and scenes
* Add fields to gallery page
* Internationalisation
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add InteractiveSpeed to scene model
* add InteractiveHeatmapSpeedGenerator
* add GenerateInteractiveHeatmapSpeedTask
* add InteractiveHeatmapSpeedTask to GenerateJob
* add InteractiveHeatmap on sceneRoutes
* delete heatmap when scene is destroyed
* render interactive heatmap in GridCard
* render InteractiveSpeed on SceneCard
* render InteractiveSpeed in SceneFileInfoPanel
* InteractiveSpeed filters
* Added joinType to join struct
* Added addInnerJoin function to perform INNER JOIN type of joins
* Added innerJoin function to perform INNER JOIN type of joins
* Use inner joins when querying images in a gallery
* Renamed addJoin to addLeftJoin
* Move scan options out of dialog
* Move autotag and clean options out of dialogs
* Move generate options out of dialog
* Animate button while tasks running
* Revert to earlier Tasks UI iteration
* Rearrange and clarify scan options
* Support a maxAge input on metadata scans.
Extend the GraphQL world with a Duration scalar. It is parsed as a
typical Go duration, i.e., "4h" is 4 hours. Alternatively, one can
pass an integer which is interpreted as seconds.
Extend Mutation.metadataScan(input: $input) to support a new optional
value, maxAge. If set, the scanner will exit early if the file it
is looking at has an mtime older than the cutOff point generated by
now() - maxAge
This speeds up scanning in the case where the user knows how old the
changes on disk are, by exiting the scan early if that is the case.
* Change maxAge into minModTime
Introduce a `Timestamp` scalar, so we have a scalar we control. Let
it accept three formats:
* RFC3339Nano
* @UNIX where UNIX is a unix-timestamp: seconds after 01-01-1970
* '<4h': a timestamp relative to the current server time
This scalar parses to a time.Time.
Use MinModTime in the scanner to filter out a large number of scan
analyzes by exiting the scan operation early.
* Heed the linter, perform errcheck
* Rename test vars for consistency.
* Code review: move minModTime into queuefiles
* Remove the ability to input Unix timestamps
Test failures on the CI-system explains why this is undesirable. It is
not clear what timezone one is operating in when entering a unix
timestamp. We could go with UTC, but it is so much easier to require an
RFC3339 timestamp, which avoids this problem entirely.
* Move the minModTime field into filters
Create a new filter input object for metadata scans, and push the
minModTime field in there. If we come up with new filters, they can
be added to that input object rather than cluttering the main input
object.
* Use utils.ParseDateStringAsTime
Replace time.Parse with utils.ParseDateStringAsTime
While here, add some more test cases for that parser.
* Push scrapeByURL into scrapers
Replace ScrapePerfomerByURL, ScrapeMovie..., ... with ScrapeByURL in
the scraperActionImpl interface. This allows us to delete a lot of
repeated code in the scrapers and replace the central part with a
switch on the scraper type.
* Fold name scraping into one call
Follow up on scraper refactoring. Name scrapers use the same code path.
This allows us to restructure some code and kill some functions, adding
variance to the name scraping code. It allows us to remove some code
repetition as well.
* Do not export loop refs.
* Simplify fragment scraping
Generalize fragment scrapers into ScrapeByFragment. This simplifies
fragment code flows into a simpler pathing which should be easier
to handle in the future.
* Eliminate more context.TODO()
In a number of cases, we have a context now. Use the context rather than
TODO() for those cases in order to make those operations cancellable.
* Pass the context for the stashbox scraper
This removes all context.TODO() in the path of the stashbox scraper,
and replaces it with the context that's present on each of the paths.
* Pass the context into subscrapers
Mostly a mechanical update, where we pass in the context for
subscraping. This removes the final context.TODO() in the scraper
code.
* Warn on unknown fields from scripts
A common mistake for new script writers are that they return fields
not known to stash. For instance the name "description" is used rather
than "details".
Decode disallowing unknown fields. If this fails, use a tee-reader to
fall back to the old behavior, but print a warning for the user in this
case. Thus, we retain the old behavior, but print warnings for scripts
which fails the more strict unknown-fields detection.
* Nil-check before running the postprocessing chain
Fixes panics when scraping returns nil values.
* Lift nil-ness in post-postprocessing
If the struct we are trying to post-process is nil, we shouldn't
enter the postprocessing flow at all. Pass the struct as a value
rather than a pointer, eliminating nil-checks as we go. Use the
top-level postProcess call to make the nil-check and then abort there
if the object we are looking at is nil.
* Allow conversion routines to handle values
If we have a non-pointer type in the interface, we should also convert
those into ScrapedContent. Otherwise we get errors on deprecated
functions.
* Add scan dialog
* Add Auto Tag dialog
* Refactor and combine Generate dialog
* Add clean dialog
* Support scan task default setting
* Support saving auto tag defaults
* Support for generate defaults
* Simplify scraper listing
Introduce an enum, scraper.Kind, which explains what we are looking
for. Make it possible to match this from a scraper struct.
Use the enum to rewrite all the listing code to use the same code path.
* Use a map, nitpick ScrapePerformerList
Let the cache store a map from ID of a scraper to the scraper. This
improves lookups when there are many scrapers, making it practically
O(1) rather than O(n). If many scrapers are stored, this is faster.
Since range expressions work unchanged, we don't have to change much,
and things will still work.
make Kind a Stringer
Rename ScraperPerformerList -> ScraperPerformerQuery since that name
is used in the other scrapers, and we value consistency.
Tune ScraperPerformerQuery:
* Return static errors
* Use the new functionality
* When loading scrapers, do so directly
Rather than first walking the directory structure to obtain file paths,
fold the load directly in the the filepath walk. This makes the code
for more direct.
* Use static ErrNotFound
If a scraper isn't found, return one static error. This paves the way
for eventually doing our own error-presenter in gqlgen.
* Store the cache in the Resolver state
Putting the scraperCache directly in the resolver avoids the need to
call manager.GetInstance() all over the place to get access to the
scraper cache. The cache is stored by pointer, so it should be safe,
since the cache will just update its internal state rather than being
overwritten.
We can now utilize the resolver state to grab the cache where needed.
While here, pass context.Context from the resolver down into a function,
which removes a context.TODO()
* Introduce ScrapedContent
Create a union in the GraphQL schema for all scraped content. This
simplifies the internal implementation because we get variance on
the output content type.
Introduce a new type ScrapedContentType which signifies the scraped
content you want as a caller.
Use these to generalize the List interface and the URL scraping
interface.
* Simplify the scraper API
Introduce a new interface for scraping. This interface is then
used in the upper half of the scraper code, to make the code use one
code flow rather than multiple code flows. Variance is currently at
the old scraper structure.
Add extending interfaces for the different ways of invoking scrapes.
Use interface conversions to convert a scraper from the cache to a
scraper supporting the extra methods.
The return path returns models.ScrapedContent.
Write a general postProcess function in the scraper, handling all
ScrapedContent via type switching. This consolidates all postprocessing
code flows.
Introduce marhsallers in the resolver code for converting ScrapedContent
into the underlying concrete types. Use this to plug the existing
fields in the Query resolver, so everything still works.
* ScrapedContent: add more marshalling functions
Handle all marshalling of ScrapedContent through marhsalling functions.
Removes some hand-rolled early variants of it, and replaces it with
a canonical code flow.
* Support loadByName via scraper_s
In order to temporarily plug a hole in the current implementation, we
use the older implementation as a hook to get the newer implementation
to run.
Later on, this can serve as a guide for how to implement the lower level
bits inside the scrapers themselves. For now, it just enables support.
* Plug the remaining scraper functions for now
Since we would like to have a scraper which works in between refactors,
plug the lower level parts of the scraper for now. It avoids us having
to tackle this part just yet.
* Move postprocessing to its own file
There's enough postprocessing to clutter the main scrapers.go file.
Move all of this into a new file, postprocessing to make the API
simpler. It now lives in scrapers.go.
* Scraper: Invoke API consistency
scraper.Cache.ScrapeByName -> ScrapeName
* Fix scraping scenes by URL
Simple typo. While here, also make a single marshaller nil-aware.
* Introduce scraper groups, consolidate loadByURL
Rename `scraper_s` into `group`. A group is a group of scrapers with
the same identity. This corresponds to a single YAML file for a scraper
configuration. It defines a group which supports different types of
scraping contexts.
Move config into the group, and lift txnManager and globalConfig to
the group.
Because we now return models.ScrapedContent we can use interfaces to
get variance from the different underlying scrapers. Use a type
switch for the URL matcher candidates. And then again for the scrapers.
This consolidates all URL scraping paths into one.
While here, remove the urlMatcher interface which isn't needed. Also
clean up the remaining interfaces for url scraping and delete code
which has no purpose anymore.
* Consolidate fragment scraping in one code path
While here, abide the linters checks.
* Refactor loadByFragment
Give it the same treatment as loadByURL:
Step 1: find a scraperActionImpl which works for the data.
Step 2: use that to scrape
Most of this is simple analysis on the data at hand. It can be pushed
down further in a later commit, but for now we leave it here.
* Remove configScraper, autotag is a scraper
Remove the remains of the configScraper struct. It now lives on in the
group struct. Kill the remaining interfaces from the old implementation
while here.
Remove group.specification since it can now be handled by a simple
func call to spec().
Work through the autotag scraper. It now implements the scraper
interface, so it can be used as a scraper. This also simplifies the
autotag scraper quite a bit since it doens't have to implement a number
of unsupported func calls.
* Simplify the fragment scraper flow
* Pass the context
Eliminate a round of context.TODO() in the scraper code by passing
the calling context down into the subsystem. This will gracefully
allow for termination of remote calls if the client goes away for some
reason in GraphQL requests.
* Improve listScrapers in the schema
Support lists of types we accept.
* Be graceful on nil values in conversion
Supporting nil-values make the API more robust in the
case of partial results in a multi-scrape situation.
* Improve listScrapers: output at-most-once
Use the ID of a scraper to reduce the output set. If a scraper has
been included, don't include it again.
* Consolidate all API level errors into resolver.go
* Reorder files and functions:
scrapers.go -> cache.go:
It almost contains nothing but the cache code.
Move errors into scraper.go from here because
It is a better place to have them living right now
group.go:
All of the group structure. This can now go from
scraper.go, making it more lean. Move group create
from config_scraper to here.
config.go:
Move the `(c config) spec()` call to here.
config_scraper.go:
Empty file by now
* Name-update the scraper interfaces
Use 'via' rather than 'loadBy'.
The scrape happens via a given scrape method, so I think this is a nice
name for it.
* Rename scrapers for consistency.
While here, improve the error formatting, so different errors come
back differently.
* Nuke the freeones field from the GraphQL schema
* Fix autotag interfacing, refactor
The autotag scraper uses a pointer receiver, but the rest of the code
we use for scraping doesn't expect a pointer-receiver. Hence, to fix
the autotag scraper, we change it to be a value receiver, like the
rest of the code.
Fix: viaScene, and viaGallery.
While here, remove a couple of pointer-receiver methods which can be
trivially rewritten into plain functions.
* Protect against pointer interfaces
The underlying code can be a bit inconsistent in what it returns.
Introduce pointer-types in the postprocessing layer and handle them
accordingly for now. Once a better understanding of the lower levels
are understood, we can lift this.
* Move ErrConversion into the models package.
The conversion error pertains to the logic of converting models.
Because of this, it should move there, so it is centralized.
* Be consistent in scraper resolver error handling
If we have a static error
Err = errors.New(..)
Then use it wrapped at the start:
fmt.Errorf("%w: ...context...", Err)
This reads better.
While here, avoid using the underlying Atoi errors: they are verbose,
and like 99% of the time, the user know what is wrong from the input
string, so just give that back.
Also, remove the scraper id from the error contexts: it is implicit,
and the error wouldn't change if we used a different scraper, which
the error message would imply.
* Mark the list*Scrapers() API as deprecated
The same functionality is now present in listScrapers.
* Improve error formatting
Think about how each error is going to be used and tweak them to be
nicer.
* Return a sorted list of scrapers
This helps testing, it's closer to what we had, caches like stable data,
and it is easier for humans. It also makes the output stable, because
map iteration is randomized.
* Fix listScrapers calls to return in ID-order
Since we need the ordering to be by ID in all situations, it is easier
to just generalize the cache listScrapers call to support multiple
scraper types.
This avoids a de-dupe map up the chain, since every scraper is only
considered once. Sorting now happens in the cache listScrapers call.
Use this generalized function in all resolvers, which are now simple
passthroughs.
* Remove UpdateConfig from the scraper cache.
This isn't needed, so get rid of it.
* Pull a context into identify
Scraping scenes in the identify tasks now use a context from up the
call chain.
* Do not store the scraper cache in the resolver.
Scraper caches are updated through
manager.singleton•RefreshScraperCache, so we can't keep a pointer to
it in the resolver. Instead, solve this by adding a fetcher method to
the resolver type. This keeps it local to the resolver, while handling
the problem of updating caches in the configuration.
* Fix incorrect tense in toast
* Rename create_entity to emphasize past tense
* Localize "Started XXX" toasts
* Localize new zh-tw texts
* Refactor "continue" into the "actions" group
* Separate overrides from config
* Don't allow changing overridden value
* Write default host and port to config file
* Use existing library value. Hide generated if set
* Support Is (not) null for all multi criterions
Add support for the Is null and Is not null modifiers for all cases of
the MultiCriterionInput and HierarchicalMultiCriterionInput. This
partially overlaps the "X Count" filter which sometimes is available
(because it would be the same as "X Count equals 0" and "X Count greater
than 0") but this also enables it for other criterions like the "Parent
Studio" filter for studios or just the "Studios" filter for scenes /
images / galleries, the "Movies" filter for scenes etc.
* Don't crash UI on bad saved filter
* Add missing code for tag parent/child
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* update tag hierarchy validation
* refactor MergeHierarchy
* update tag hierarchy error message
* rename tag hierarchy function
* add tag path to error message
* Rename EnsureHierarchy to ValidateHierarchy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix responsive layout
* Refactor MainNavbar
* Stick the navbar to the bottom on mobile
* Fix menu item icon-text vertical alignment
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add delete file and generated files by default config options
* add alert message with files to be deleted
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Split performerCreate page into separate page
* Split studioCreate into a separate page
* Remove Partial types from performer/studio
* Split tagCreate into a separate page
* Split movieCreate into a separate page
* Split out galleryCreate into its own page
* Add loader to scene page
* Fix performer name fallback
* Fix movie layout shift
* Fix prompt comment and switch studio prompt to localized string
The version checking code performs its own error management and will
not pass errors to the caller. Hence, it needs to be aware of the types
of errors which can be returned.
In particular, the context.Canceled error will be returned if the
context is aborted through cancelation. This happens when the request
is terminated by tapping CTRL-C or if the browser request is terminated
while we are sitting waiting for the GH API.
* Docker CI builds: half the size, less than half the build time
* Add an "Official Build" Designator
* Fix .git constantly invalidating build cache, use distro ffmpeg
* Fix official build detection, add some compiler image docs
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make copies of buffers
Avoid reusing one of the incoming arrays as a append extension, and
make a copy of the data. It's cleaner in the long run and possibly
easier for the GC to maintain.
* Avoid appendAssign problems in tag code
Reuse the existing slice when appending.
* Fix appendAssign in encoder_scene_preview_chunk
Appending and creating a new slice is somewhat unintuitive since the
underlying slice might be extended to satisfy the new capacity. This
sometimes leads to faulty logic.
Rewrite the code so it reuses `args` for all appending, and builds one
array clearly in the code. It follows the general style of the function
where `args` is being built in small incremental batches and avoids
the introduction of new names.
* Enable the appendAssign check
This makes us pass all gocritic warnings.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add Cookies directly to the request
Rather than maintaining a cookie jar on a one-shot HTTP client, maintain
the jar ourselves: make a new jar, then use it to select the right
cookies.
The cookies are set on the request rather than on the client. This will
retain the current behavior as we are always throwing the client away
after each use.
This patch enables the lifting of the http client as well over time.
* Introduce a cached scraper HTTP client
The scraper cache is augmented with an *http.Client. These are safe for
concurrent use, so the pointer can safely be passed around. Push this
into scraper configurations where applicable, next to the txnManagers.
When we issue a loadUrl request, do so on the cached *http.Client,
which will reuse existing idle connections in the client if any are
present.
* Set MaxIdleConnsPerHost. Closes#1850
We allow for up to 8 idle connections to a single host. This should
make concurrent operation toward the same host reuse connections, even
for sizeable concurrency.
The number isn't bumped excessively high. We should probably limit
concurrency toward a single site anyway, since we'll be able to overrun
a site with queries quite easily if we have many concurrent goroutines
issuing requests at the same time.
* Reinstate driverOptions / useCDP check
Use DeMorgan's laws to invert the logic and exit early. Fixes tests
breaking.
* Documentation fixup.
* Use the scraper http.Client when fetching images
Fold image fetchers onto the cached scraper http.Client as well. This
makes the scraper have a single http.Client cache for all its
operations.
Thread the client upwards to the relevant attachment points: either the
cache, or a stash_box instance, which is extended to include a pointer
to the client.
Style roughly follows that of txnManagers.
* Use the same http Client as the GraphQL client use
Rather than using http.DefaultClient, use the same client as the
GraphQL client use in the stash_box subsystem. This localizes the
client used in the subsystem into the constructing New.. call.
* Hoist HTTP client construction
Create a function for initializaing the HTTP Client we use. While here
hoist magic numbers into constants. Introduce a proper static redirect
error and use it in the client code as well.
* Reinstate printCookies
This is a debugging function, and it might still come in handy in the
future at some point.
* Nitpick comment.
* Minor tidy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add a space after // comments
For consistency, the commentFormatting lint checker suggests a space
after each // comment block. This commit handles all the spots in
the code where that is needed.
* Rewrite documentation on functions
Use the Go idiom of commenting:
* First sentence declares the purpose.
* First word is the name being declared
The reason this style is preferred is such that grep is able to find
names the user might be interested in. Consider e.g.,
go doc -all pkg/ffmpeg | grep -i transcode
in which case a match will tell you the name of the function you are
interested in.
* Remove old code comment-blocks
There are some commented out old code blocks in the code base. These are
either 3 years old, or 2 years old. By now, I don't think their use is
going to come back any time soon, and Git will track old pieces of
deleted code anyway.
Opt for deletion.
* Reorder imports
Split stdlib imports from non-stdlib imports in files we are touching.
* Use a range over an iteration variable
Probably more go-idiomatic, and the code needed comment-fixing anyway.
* Use time.After rather than rolling our own
The idiom here is common enough that the stdlib contains a function for
it. Use the stdlib function over our own variant.
* Enable the commentFormatting linter
* Don't capitalize local variables
ValidCodecs -> validCodecs
* Capitalize deprecation markers
A deprecated marker should be capitalized.
* Use re.MustCompile for static regexes
If the regex fails to compile, it's a programmer error, and should be
treated as such. The regex is entirely static.
* Simplify else-if constructions
Rewrite
else { if cond {}}
to
else if cond {}
* Use a switch statement to analyze formats
Break an if-else chain. While here, simplify code flow.
Also introduce a proper static error for unsupported image formats,
paving the way for being able to check against the error.
* Rewrite ifElse chains into switch statements
The "Effective Go" https://golang.org/doc/effective_go#switch document
mentions it is more idiomatic to write if-else chains as switches when
it is possible.
Find all the plain rewrite occurrences in the code base and rewrite.
In some cases, the if-else chains are replaced by a switch scrutinizer.
That is, the code sequence
if x == 1 {
..
} else if x == 2 {
..
} else if x == 3 {
...
}
can be rewritten into
switch x {
case 1:
..
case 2:
..
case 3:
..
}
which is clearer for the compiler: it can decide if the switch is
better served by a jump-table then a branch-chain.
* Rewrite switches, introduce static errors
Introduce two new static errors:
* `ErrNotImplmented`
* `ErrNotSupported`
And use these rather than forming new generative errors whenever the
code is called. Code can now test on the errors (since they are static
and the pointers to them wont change).
Also rewrite ifElse chains into switches in this part of the code base.
* Introduce a StashBoxError in configuration
Since all stashbox errors are the same, treat them as such in the code
base. While here, rewrite an ifElse chain.
In the future, it might be beneifical to refactor configuration errors
into one error which can handle missing fields, which context the error
occurs in and so on. But for now, try to get an overview of the error
categories by hoisting them into static errors.
* Get rid of an else-block in transaction handling
If we succesfully `recover()`, we then always `panic()`. This means the
rest of the code is not reachable, so we can avoid having an else-block
here.
It also solves an ifElse-chain style check in the code base.
* Use strings.ReplaceAll
Rewrite
strings.Replace(s, o, n, -1)
into
strings.ReplaceAll(s, o, n)
To make it consistent and clear that we are doing an all-replace in the
string rather than replacing parts of it. It's more of a nitpick since
there are no implementation differences: the stdlib implementation is
just to supply -1.
* Rewrite via gocritic's assignOp
Statements of the form
x = x + e
is rewritten into
x += e
where applicable.
* Formatting
* Review comments handled
Stash-box is a proper noun.
Rewrite a switch into an if-chain which returns on the first error
encountered.
* Use context.TODO() over context.Background()
Patch in the same vein as everything else: use the TODO() marker so we
can search for it later and link it into the context tree/tentacle once
it reaches down to this level in the code base.
* Tell the linter to ignore a section in manager_tasks.go
The section is less readable, so mark it with a nolint for now. Because
the rewrite enables a ifElseChain, also mark that as nolint for now.
* Use strings.ReplaceAll over strings.Replace
* Apply an ifElse rewrite
else { if .. { .. } } rewrite into else if { .. }
* Use switch-statements over ifElseChains
Rewrite chains of if-else into switch statements. Where applicable,
add an early nil-guard to simplify case analysis. Also, in
ScanTask's Start(..), invert the logic to outdent the whole block, and
help the reader: if it's not a scene, the function flow is now far more
local to the top of the function, and it's clear that the rest of the
function has to do with scene management.
* Enable gocritic on the code base.
Disable appendAssign for now since we aren't passing that check yet.
* Document the nolint additions
* Document StashBoxBatchPerformerTagInput
* Use the request context
The code uses context.Background() in a flow where there is a
http.Request. Use the requests context instead.
* Use a true context in the plugin example
Let AddTag/RemoveTag take a context and use that context throughout
the example.
* Avoid the use of context.Background
Prefer context.TODO over context.Background deep in the call chain.
This marks the site as something which we need to context-handle
later, and also makes it clear to the reader that the context is
sort-of temporary in the code base.
While here, be consistent in handling the `act` variable in each
branch of the if .. { .. } .. check.
* Prefer context.TODO over context.Background
For the different scraping operations here, there is a context
higher up the call chain, which we ought to use. Mark the call-sites
as TODO for now, so we can come back later on a sweep of which parts
can be context-lifted.
* Thread context upwards
Initialization requires context for transactions. Thread the context
upward the call chain.
At the intialization call, add a context.TODO since we can't break this
yet. The singleton assumption prevents us from pulling it up into main for
now.
* make tasks context-aware
Change the task interface to understand contexts.
Pass the context down in some of the branches where it is needed.
* Make QueryStashBoxScene context-aware
This call naturally sits inside the request-context. Use it.
* Introduce a context in the JS plugin code
This allows us to use a context for HTTP calls inside the system.
Mark the context with a TODO at top level for now.
* Nitpick error formatting
Use %v rather than %s for error interfaces.
Do not begin an error strong with a capital letter.
* Avoid the use of http.Get in FFMPEG download chain
Since http.Get has no context, it isn't possible to break out or have
policy induced. The call will block until the GET completes. Rewrite
to use a http Request and provide a context.
Thread the context through the call chain for now. provide
context.TODO() at the top level of the initialization chain.
* Make getRemoteCDPWSAddress aware of contexts
Eliminate a call to http.Get and replace it with a context-aware
variant.
Push the context upwards in the call chain, but plug it before the
scraper interface so we don't have to rewrite said interface yet.
Plugged with context.TODO()
* Scraper: make the getImage function context-aware
Use a context, and pass it upwards. Plug it with context.TODO()
up the chain before the rewrite gets too much out of hand for now.
Minor tweaks along the way, remove a call to context.Background()
deep in the call chain.
* Make NOTIFY request context-aware
The call sits inside a Request-handler. So it's natural to use the
requests context as the context for the outgoing HTTP request.
* Use a context in the url scraper code
We are sitting in code which has a context, so utilize it for the
request as well.
* Use a context when checking versions
When we check the version of stash on Github, use a context. Thread
the context up to the initialization routine of the HTTP/GraphQL
server and plug it with a context.TODO() for now.
This paves the way for providing a context to the HTTP server code in a
future patch.
* Make utils func ReadImage context-aware
In almost all of the cases, there is a context in the call chain which
is a natural use. This is true for all the GraphQL mutations.
The exception is in task_stash_box_tag, so plug that task with
context.TODO() for now.
* Make stash-box get context-aware
Thread a context through the call chain until we hit the Client API.
Plug it with context.TODO() there for now.
* Enable the noctx linter
The code is now free of any uncontexted HTTP request. This means we
pass the noctx linter, and we can enable it in the code base.
* Add collation to directory listings. Closes#1806
Introduce a new `locale` arg to the `Query.directory` field. Set "en"
as the default for the field for backward compatibility. Use the given
locale, sending it through a language matcher, and use `x/text` as the
collation engine for the matched language.
Augment the file `ListDirs` call to optionally take a Collator. If the
Collator is given, sort file listings according to the collators rules.
While here, document the GraphQL schema a bit more.
Add matchers by looking at the current front-end locales, and make sure
each of these occur in the matcher list.
* Language matcher touchups
* Avoid having `en-US` twice.
* Introduce `en-AU`.
* Pass IgnoreCase and Numeric collation
Allow the collator to be configured with options. Pass the options
IgnoreCase and Numeric to the collator.
* Replace error assertions with Go 1.13 style
Use `errors.As(..)` over type assertions. This enables better use of
wrapped errors in the future, and lets us pass some errorlint checks
in the process.
The rewrite is entirely mechanical, and uses a standard idiom for
doing so.
* Use Go 1.13's errors.Is(..)
Rather than directly checking for error equality, use errors.Is(..).
This protects against error wrapping issues in the future.
Even though something like sql.ErrNoRows doesn't need the wrapping, do
so anyway, for the sake of consistency throughout the code base.
The change almost lets us pass the `errorlint` Go checker except for
a missing case in `js.go` which is to be handled separately; it isn't
mechanical, like these changes are.
* Remove goconst
goconst isn't a useful linter in many cases, because it's false positive
rate is high. It's 100% for the current code base.
* Avoid direct comparison of errors in recover()
Assert that we are catching an error from recover(). If we are,
check that the error caught matches errStop.
* Enable the "errorlint" checker
Configure the checker to avoid checking for errorf wraps. These are
often false positives since the suggestion is to blanket wrap errors
with %w, and that exposes the underlying API which you might not want
to do.
The other warnings are good however, and with the current patch stack,
the code base passes all these checks as well.
* Configure rowserrcheck
The project uses sqlx. Configure rowserrcheck to include said package.
* Mechanically rewrite a large set of errors
Mechanically search for errors that look like
fmt.Errorf("...%s", err.Error())
and rewrite those into
fmt.Errorf("...%v", err)
The `fmt` package is error-aware and knows how to call err.Error()
itself.
The rationale is that this is more idiomatic Go; it paves the
way for using error wrapping later with %w in some sites.
This patch only addresses the entirely mechanical rewriting caught by
a project-side search/replace. There are more individual sites not
addressed by this patch.
Reduce allocations. Don't create intermediary arrays which we then
consume right after. Manually fuse the arrays and decode straight into
the sum instead.
Furthermore, don't invoke a Reader, but carve out the locations via a
loop, directly.
These two changes taken together speeds up oshash computations by a
factor of 10 according to the benchmark tests. The main reason for
this change is a much lowered memory allocation rate which in turn
improves GC pressure.
While here, add a benchmark for oshash computations and use it for
testing the performance.
* Refactor scraper structures
* Move matching code into new package
* Add autotag scraper
* Always check first letter of auto-tag names
* Account for nulls
Co-authored-by: Kermie <kermie@isinthe.house>
* Add security against publicly exposed services
* Add trusted proxies setting, validate proxy chain against internet access
* Validate chain on local proxies too
* Move authentication handler to separate file
* Add startup check and log if tripwire is active
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Enable safe linters
Enable the linters dogsled, rowserrcheck, and sqlclosecheck.
These report no errors currently in the code base.
Enable misspell.
Misspell finds two spelling mistakes in comments, which are fixed by the
patch as well.
Add and sort linters which are relatively
safe to add over time. Comment them out for now.
* Close the response body
If we can get a HTTP response, it has a body which ought to be closed.
By doing so, we avoid potentially leaking connections.
* Enable the exportloopref linter
There are two places in the code with these warnings. Fix them while
enabling the linter.
* Remove redundant types in tests
If a slice already determines the type, the inner type declaration is
redundant. Remove the inner declarations.
* Mark autotag test cases as parallel
Autotag test cases is by far the outlier when it comes to test time.
While go test runs test cases in parallel,
it doesn't do so inside a given package, unless one marks the test cases
as parallel.
This change provides a significant speedup on a 8-core machine for test
runs.
* Rewrite lightbox code
* Don't render offscreen images
* Scroll up to zoom in
* Support touch gestures
* Add reset zoom button
* Align top of image on original/fit horizontal
* Add scrollmode setting
* Add scale up option
* Add option to maintain zoom when transisitioning
* Fix image slideshow wrap around
* Wrap around on previous on first page/image
* Fix single page issues
* Fix two-pointer zoom mode incorrectly activated
* Add API support for filtering tags by parent / children
* Add parent & child tags filters for tags to UI
* Add API support for filtering tags by parent / child count
* Add parent & child count filters for tags to UI
* Update db generator
* Add missing build tag
* Add unit tests
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix all revive warnings in the code base
All of these are of the form
```
var Identifier Type = Expr
```
where the `Type` is known from the output of `Expr` and can be omitted
as a result.
* Handle unchecked errors
* Remove new-from-rev
Since the project passes all linter checks now, including older
revisions, we can remove new-from-rev. While here, reorder the linter
config file, and move the enabled linters up and settings down.
* Fix failing test cases
Studio & Performer export tests use local time rather than UTC. This
fixes the test cases so integration test
passes.
When the golangci workflow action was added, we moved large parts of
validation to the parallel action. However, integration testing still
has to happen on the main build action, as it isn't covered by the
validation action.
Reenable the `it` target through docker container build.
This fixes backend test errors sneaking through the build action.
* Add movies and movie_count properties to Performer type
Extend the GraphQL API to allow getting the movies and movie count by
performer.
* Add movies count to performer card
* Add movies and movie_count properties to Studio type
Extend the GraphQL API to allow getting the movies and movie count by
studio.
* Add movies count to studio card
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Add golangci-lint workflow
* Add a bit more lenient linter timeout
1 Minute isn't always enough, so bump to 3.
* Document golangci, add make target
Document how to get golangci-lint in the README file. While here,
provide a QOL target in the makefile
for the linter, and make it part of validation.
* Introduce .golangci.yml
This is the default golangci-lint configuration file location. Use it.
Move configuration into the yaml file, and enable the default set of
linters; we know we pass most of those.
* Add gofmt and revive to golangci-lint
Read the golangci-lint source code to figure out the configuration format.
Copy the configuration from `revive.toml` into the linter configuration.
* Do not set simplify on gofmt
The project currently runs without simplify. So for consistency, don't
make that a requirement for the linter.
* Add new-from-rev
Older issues should not be considered a failure for new PRs and issues.
Use new-from-from to make the current develop as the point-in-time for
when we consider errors.
Once in the tree, we can go and fix the older errors in separate
patches, taking a little bit at a time.
* Move to golangci-lint
Rewrite the way we run targets in the makefile, so it is split between
frontend and backend.
Use the frontend build steps in build.yml
Update README to reflect the new world order.
* Remove check-gofmt.sh
The tool now runs as part of golangci-lint, in particular through the
'validate' target in the Makefile.
* Remove targets for golangci-lint
Fold these targets into the `lint` target. While here, update README.
* Support getting scenes on movies in the API
* Make movie card visually consistent with scene etc
* Add date
* Add synopsis
* Show scene count with hover listing the scenes
* Move scene index to button
* Move scene number to own section
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix sort direction being lost filtering scenes list
Fix thevsort direction being lost on the scenes (& images & galleries?)
lists when filtering the list by using the "badges" on the tags /
performers / ... popovers.
* Fix using back button not clearing filters to empty
Fix (re)setting of a lists criterion based on the URL in case the new
URL doesn't have criterions / "c" set. When it's unset it should be
assumed the criterions must be empty, instead of maintaining the current
criterions (which could be populated wirh the old criterions when the
list was shown before).
* Set page to 1 on list when applying new filter
* Add date & details to gallery card
Make the gallery card visually consistent with the scenes card. So move
the "X images" to a button at the bottom (where tags, performers and
scenes are shown as well) and add the date & details to the card.
* Log 3 unchecked errors
Rather than ignore errors, log them at
the WARNING log level.
The server has been functioning without these, so assume they are not at
the ERROR level.
* Log errors in concurrency test
If we can't initialize the configuration, treat the test as a failure.
* Undo the errcheck on configurations for now.
* Handle unchecked errors in pkg/manager
* Resolve unchecked errors
* Handle DLNA/DMS unchecked errors
* Handle error checking in concurrency test
Generalize config initialization, so we can initialize a configuration
without writing it to disk.
Use this in the test case, since otherwise the test fails to write.
* Handle the remaining unchecked errors
* Heed gosimple in update test
* Use one-line if-initializer statements
While here, fix a wrong variable capture error.
* testing.T doesn't support %w
use %v instead which is supported.
* Remove unused query builder functions
The Int/String criterion handler functions are now generalized.
Thus, there's no need to keep these functions around anymore.
* Mark filterBuilder.addRecursiveWith nolint
The function is useful in the future and no other refactors are looking
nice.
Keep the function around, but tell the linter to ignore it.
* Remove utils.Btoi
There are no users of this utility function
* Return error on scan failure
If we fail to scan the row when looking for the
unique checksum index, then report the error upwards.
* Fix comments on exported functions
* Fix typos
* Fix startup error
* Improve image scanning performance and thumbnail generation
* Add vips-tools to build image
* Add option to write generated thumbnails to disk
* Fallback to image if thumbnail generation fails
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
Rather than passing a pointer to a waitgroup into task.Start(..)
functions, handle the waitgroup.Done() at the callsite.
This makes waitgroup handling local to its definition rather than it
being spread out over multiple files. Tasks now simply execute, and
the policy of waiting on them is handled by the caller.
* Add indexes for path and checksum to images
The scenes table has unique indexes/constraints on path and checksum
colums. The images table doesn't, which doesn't really make sense, as
scanning uses these colums extensively which warrents an index, and both
should be unique as well.
Adding these indexes thus heavily improves the scanning tasks
performance. On a database containing 4700 images a (re)scan of those
4700 files, which thus shouldn't do anything, took 1.2 seconds, with the
indexes added this only takes 0.4 seconds. Taking the same test on a
generated database containing 4M images + the actual 4700 images took 26
minutes for a rescan, and with the index existing also only takes 0.4
seconds.
* Add images.checksum unique constraint in code with fallback
Work around the issue where in some cases duplicate images (/checksums
on images) might exist. This as discussed in #1740 by creating the index
on startup and in case of an error logging the duplicates. This so the
users where this scenario exists can correct the database (by searching
on the logged checksum(s) and removing the duplicates) and after a
restart the unique index / constraint will still be created. In case
when creating the unique index fails a "normal" / non-unique index is
created as surrogate so the user will still get the performance benefit
(for example during scanning) without being forced to remove the
duplicates and restart beforehand. This surrogate is also automatically
cleaned up after the unique index is succesfully created.
NORMAL sync is safe when using WAL journaliing.
It cuts the amount of sync calls to the disk, resulting in faster write
operation. On power loss, the database will perhaps lose some ongoing
commits, but that is all.
The expectation is that if the database lives on the same disk as the
stash, this could help performance quite a bit under heavier operation.
* Avoid redundant logging in migrations
Return the error and let the caller handle the logging of the error if
needed.
While here, defer m.Close() to the function boundary.
* Treat errors as values
Use %v rather than %s and pass the errors directly.
* Generate a wrapped error on stat-failure
* Log 3 unchecked errors
Rather than ignore errors, log them at
the WARNING log level.
The server has been functioning without these, so assume they are not at
the ERROR level.
* Propagate errors upward
Failure in path generation was ignored. Propagate the errors upward the
call stack, so it can be handled at the level of orchestration.
* Warn on errors
Log errors rather than quenching them.
Errors are logged at the Warn-level for now.
* Check error when creating test databases
Use the builtin log package and stop the program fatally on error.
* Add warnings to uncheck task errors
Focus on the task system in a single commit, logging unchecked
errors as warnings.
* Warn-on-error in API routes
Look through the API routes, and make sure errors are being logged if
they occur. Prefer the Warn-log-level because none of these has proven
to be fatal in the system up until now.
* Propagate error when adding Util API
* Propagate error on adding util API
* Return unhandled error
* JS log API: propagate and log errors
* JS Plugins: log GQL addition failures.
* Warn on failure to write to stdin
* Warn on failure to stop task
* Wrap viper.BindEnv
The current viper code only errors if no name is provided, so it should
never fail. Rewrite the code flow to factor through a panic-function.
This removes error warnings from this part of the code.
* Log errors in concurrency test
If we can't initialize the configuration, treat the test as a failure.
* Warn on errors in configuration code
* Plug an unchecked error in gallery zip walking
* Warn on screenshot serving failure
* Warn on encoder screenshot failure
* Warn on errors in path-handling code
* Undo the errcheck on configurations for now.
* Use one-line initializers where applicable
rather than using
err := f()
if err!= nil { ..
prefer the shorter
if err := f(); err != nil { ..
If f() isn't too long of a name, or wraps a function with a body.
* Execute Gallery.Create.Post plugin hook during scan
Fix issue where Gallery.Create.Post hook is not executed when a new
gallery is created during scan, when the gallery is created based on the
folder.
* Fix Gallery.Create.Post hook being invoked in transaction
Invoke the Gallery.Create.Post hook during zip scan after the
transaction has been committed. This is necessary to allow the plugin to
access the gallery (using GraphQL API). Otherwise the API obviously uses
a different database transaction which can't find the gallery as it
isn't committed yet.
* Fix logs from scraper and plugins not being shown in UI
Using `logger.` in the logger package to write logs is "incorrect". This
as the package contains a variable named `logger` which contains the
logrus instance. So instead of the log line being handled by the custom
log implementation / wrapper which makes sure the lines are shown in the
UI as well, it's written to logrus directly meaning the wrapper is
skipped.
This "issue" is obviously triggered because in any other place
`logger.X` can be used and it will used the custom logger package /
wrapper which works fine.
* Add plugin / scraper name to logging output
Indicate which plugin / scraper wrote a log message by including its
name to the `[Scrape]` prefix.
* Add missing addLogItem call
* Generate screenshot images for markers
In some scenarios it might not be possible to use the preview video or
image of markers, i.e. when only static images are supported like in
Kodi. So generate a static screenshot as well.
* Make generating animated and static image optional for markers
* Use screenshot for markers when preview type is set to static image
Add some form of backwards compatibility in the UI for
`IHierarchicalLabeledIdCriterion`. With this backwards compatibility
it's possible to recall saved filters which use the tags filter.
Otherwise stuff would blow up because the saved filter has a different
structure than what the `IHierarchicalLabeledIdCriterion` expects.
* Add migration to create studio aliases table
* Refactor studioQueryBuilder.Query to use filterBuilder
* Expand GraphQL API with aliases support for studio
* Add aliases support for studios to the UI
* List aliases in details panel
* Allow editing aliases in edit panel
* Add 'aliases' filter when searching
* Find studios by alias in filter / select
* Add auto-tagging based on studio aliases
* Support studio aliases for filename parsing
* Support importing and exporting of studio aliases
* Search for studio alias as well during scraping
* Add migration script for tag relations table
* Expand hierarchical filter features
Expand the features of the hierarchical multi input filter with support
for using a relations table, which only has parent_id and child_id
columns, and support adding an additional intermediate table to join on,
for example for scenes and tags which are linked by the scenes_tags
table as well.
* Add hierarchical filtering for tags
* Add hierarchical tags support to scene markers
Refactor filtering of scene markers to filterBuilder and in the process
add support for hierarchical tags as well.
* List parent and child tags on tag details page
* Support setting parent and child tags
Add support for setting parent and child tags during tag creation and
tag updates.
* Validate no loops are created in tags hierarchy
* Update tag merging to support tag hierarcy
* Add unit tests for tags.EnsureUniqueHierarchy
* Fix applying recursive to with clause
The SQL `RECURSIVE` of a `WITH` clause only needs to be applied once,
imediately after the `WITH`. So this fixes the query building to do just
that, automatically applying the `RECURSIVE` keyword when any added with
clause is added as recursive.
* Rename hierarchical root id column
* Rewrite hierarchical filtering for performance
Completely rewrite the hierarchical filtering to optimize for
performance. Doing the recursive query in combination with a complex
query seems to break SQLite optimizing some things which means that the
recursive part might be 2,5 second slower than adding a static
`VALUES()` list. This is mostly noticable in case of the tag hierarchy
where setting an exclusion with any depth (or depth: all) being applied
has this performance impact of 2,5 second. "Include" also suffered this
issue, but some rewritten query by joining in the *_tags table in one
pass and applying a `WHERE x IS NOT NULL` filter did seem to optimize
that case. But that optimization isn't applied to the `IS NULL` filter
of "exclude". Running a simple query beforehand to get all (recursive)
items and then applying them to the query doesn't have this performance
penalty.
* Remove UI references to child studios and tags
* Add parents to tag export
* Support importing of parent relationship for tags
* Assign stable ids to parent / child badges
* Silence Apollo warning on parents/children fields on tags
Silence warning triggered by Apollo GraphQL by explicitly instructing it
to use the incoming parents/children values. By default it already does
this, but it triggers a warning as it might be unintended that it uses
the incoming values (instead of for example merging both arrays).
Setting merge to false still applies the same behaviour (use only
incoming values) but silences the warning as it's explicitly configured
to work like this.
* Rework detecting unique tag hierarchy
Completely rework the unique tag hierarchy to detect invalid hierarchies
for which a tag is "added in the middle". So when there are tags A <- B
and A <- C, you could previously edit tag B and add tag C as a sub tag
without it being noticed as parent A being applied twice (to tag C).
While afterwards saving tag C would fail as tag A was applied as parent
twice. The updated code correctly detects this scenario as well.
Furthermore the error messaging has been reworked a bit and the message
now mentions both the direct parent / sub tag as well as the tag which
would results in the error. So in aboves example it would now show the
message that tag C can't be applied because tag A already is a parent.
* Update relations on cached tags when needed
Update the relations on cached tags when a tag is created / updated /
deleted so these always reflect the correct state. Otherwise (re)opening
a tag might still show the old relations untill the page is fully
reloaded or the list is navigated. But this obviously is strange when
you for example have tag A, create or update tag B to have a relation to
tag A, and from tags B page click through to tag A and it doesn't show
that it is linked to tag B.
The strings.Replace function counts the number of replacements. If 0,
the original string is returned. Hence, there is no need to check if a
replacement will happen before doing the work.
* Remove stuff which isn't being used
Some fields, functions and structs aren't in use by the project. Remove
them for janitorial reasons.
* Remove more unused code
All of these functions are currently not in use. Clean up the code by
removal, since the version control has the code if need be.
* Remove unused functions
There's a large set of unused functions and variables in the code base.
Remove these, so it clearer what code to support going forward.
Dead code has been eliminated.
Where applicable, comment const-sections in tests, so reserved
identifiers are still known.
* Fix use-def of tsURL
The first def of tsURL doesn't matter because there's no use before
we hit the 2nd def.
* Remove dead code assignment
Setting logFile = "" is effectively dead code, because there's no use
of it later.
* Comment out found
The variable 'found' is dead in the function (because no post-process
action is following it). Comment it for now.
* Comment dead code in tests
These might provide hints as to what isn't covered at the moment.
* Dead code removal
In the case of constants where iota is involved, move the iota so it
matches the current key values.
This avoids problems with persistently stored key IDs.
* Bump Go to 1.17, refactor build/x86_64 Dockerfile to make better use of multi-stage
* Bump to 1.17 from 1.16
* Bump packr version, provide needed legacy env var
* Add apple silicon support, fix macos build chain
* Update unused travis ci
* Fix error string capitalization
Error strings often follow another string. Hence, they should not be
capitalized, unless referencing a name.
* Uncapitalize more error strings
While here, use %v on the error directly, which makes it easier to wrap
the error later with %w if need be.
* Uncapitalize more error strings
While here, rename Url to URL as a nitpick.
* When stopping, close the database
This patch is likely to cause errornous behavior in the application.
This is due to the fact that the application doesn't gracefully shut
down, but is forcefully terminated.
However, the purpose is to uncover what needs to be done, to make
it a more graceful shutdown.
The call to p.ExecuteTask happens in a separate go-routine. It writes,
under m.mutex, into the job's details. However, the test goroutine
itself reads j.Details which is a read race.
Protect the reads in the test by the lock.
This makes `package job` pass `go test -race`
* Unify scraped types
* Make name fields optional
* Unify single scrape queries
* Change UI to use new interfaces
* Add multi scrape interfaces
* Use images instead of image
[](https://github.com/stashapp/stash/releases/latest)
### **Stash is a self-hosted webapp written in Go which organizes and serves your porn.**

**Stash is a locally hosted web-based app written in Go which organizes and serves your porn.**
* It can gather information about videos in your collection from the internet, and is extensible through the use of community-built plugins for a large number of content producers.
* It supports a wide variety of both video and image formats.
*Stash gathers information about videos in your collection from the internet, and is extensible through the use of community-built plugins for a large number of content producers and sites.
* Stash supports a wide variety of both video and image formats.
* You can tag videos and find them later.
*It provides statistics about performers, tags, studios and other things.
*Stash provides statistics about performers, tags, studios and more.
You can [watch a SFW demo video](https://vimeo.com/545323354) to see it in action.
For further information you can [read the in-app manual](ui/v2.5/src/docs/en).
For further information you can consult the [documentation](https://docs.stashapp.cc) or [read the in-app manual](ui/v2.5/src/docs/en).
# Installing stash
# Installing Stash
## via Docker
#### Windows Users:
Follow [this README.md in the docker directory.](docker/production/README.md)
As of version 0.27.0, Stash doesn't support anymore _Windows 7, 8, Server 2008 and Server 2012._
The Stash server runs on macOS, Windows, and Linux. Download the [latest release here](https://github.com/stashapp/stash/releases).
Download links for other platforms and architectures are available on the [Releases page](https://github.com/stashapp/stash/releases).
Run the executable (double click the exe on windows or run `./stash-osx` / `./stash-linux` from the terminal on macOS / Linux) and navigate to either https://localhost:9999 or http://localhost:9999 to get started.
## First Run
*Note for Windows users:* Running the app might present a security prompt since the binary isn't yet signed. Bypass this by clicking "more info" and then the "run anyway" button.
#### Windows/macOS Users: Security Prompt
#### FFMPEG
On Windows or macOS, running the app might present a security prompt since the binary isn't yet signed.
If stash is unable to find or download FFMPEG then download it yourself from the link for your platform:
On Windows, bypass this by clicking "more info" and then the "run anyway" button. On macOS, Control+Click the app, click "Open", and then "Open" again.
The `ffmpeg(.exe)` and `ffprobe(.exe)` files should be placed in `~/.stash` on macOS / Linux or `C:\Users\YourUsername\.stash` on Windows.
#### FFmpeg
Stash requires FFmpeg. If you don't have it installed, Stash will download a copy for you. It is recommended that Linux users install `ffmpeg` from their distro's package manager.
# Usage
## Quickstart Guide
1) Download and install Stash and its dependencies
2) Run Stash. It will prompt you for some configuration options and a directory to index (you can also do this step afterward)
3) After configuration, launch your web browser and navigate to the URL shown within the Stash app.
Stash is a web-based application. Once the application is running, the interface is available (by default) from http://localhost:9999.
**Note that Stash does not currently retrieve and organize information about your entire library automatically.** You will need to help it along through the use of [scrapers](blob/develop/ui/v2.5/src/docs/en/Scraping.md). The Stash community has developed scrapers for many popular data sources which can be downloaded and installed from [this repository](https://github.com/stashapp/CommunityScrapers).
On first run, Stash will prompt you for some configuration options and media directories to index, called "Scanning" in Stash. After scanning, your media will be available for browsing, curating, editing, and tagging.
The simplest way to tag a large number of files is by using the [Tagger](https://github.com/stashapp/stash/blob/develop/ui/v2.5/src/docs/en/Tagger.md) which uses filename keywords to help identify the file and pull in scene and performer information from our stash-box database. Note that this data source is not comprehensive and you may need to use the scrapers to identify some of your media.
Stash can pull metadata (performers, tags, descriptions, studios, and more) directly from many sites through the use of [scrapers](https://github.com/stashapp/stash/blob/develop/ui/v2.5/src/docs/en/Manual/Scraping.md), which integrate directly into Stash. Identifying an entire collection will typically require a mix of multiple sources:
- The project maintains [StashDB](https://stashdb.org/), a crowd-sourced repository of scene, studio, and performer information. Connecting it to Stash will allow you to automatically identify much of a typical media collection. It runs on our stash-box software and is primarily focused on mainstream digital scenes and studios. Instructions, invite codes, and more can be found in this guide to [Accessing StashDB](https://guidelines.stashdb.org/docs/faq_getting-started/stashdb/accessing-stashdb/).
- Several community-managed stash-box databases can also be connected to Stash in a similar manner. Each one serves a slightly different niche and follows their own methodology. A rundown of each stash-box, their differences, and the information you need to sign up can be found in this guide to [Accessing Stash-Boxes](https://guidelines.stashdb.org/docs/faq_getting-started/stashdb/accessing-stash-boxes/).
- Many community-maintained scrapers can also be downloaded, installed, and updated from within Stash, allowing you to pull data from a wide range of other websites and databases. They can be found by navigating to Settings -> Metadata Providers -> Available Scrapers -> Community (stable). These can be trickier to use than a stash-box because every scraper works a little differently. For more information, please visit the [CommunityScrapers repository](https://github.com/stashapp/CommunityScrapers).
- All of the above methods of scraping data into Stash are also covered in more detail in our [Guide to Scraping](https://docs.stashapp.cc/beginner-guides/guide-to-scraping/).
## CLI
<sub>[StashDB](http://stashdb.org) is the canonical instance of our open source metadata API, [stash-box](https://github.com/stashapp/stash-box).</sub>
Stash runs as a command-line app and local web server. There are some command-line options available, which you can see by running `stash --help`.
For example, to run stash locally on port 80 run it like this (OSX / Linux) `stash --host 127.0.0.1 --port 80`
Stash is available in 32 languages (so far!) and it could be in your language too. We use Weblate to coordinate community translations. If you want to help us translate Stash into your language, you can make an account at [Codeberg's Weblate](https://translate.codeberg.org/projects/stash/stash/) to get started contributing new languages or improving existing ones. Thanks!
Check out our documentation on [Stash-Docs](https://docs.stashapp.cc) for information about the software, questions, guides, add-ons and more.
This command would need customizing for your environment. [This link](https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl) might be useful.
Once you have a certificate and key file name them `stash.crt` and `stash.key` and place them in the same directory as the `config.yml` file, or the `~/.stash` directory. Stash detects these and starts up using HTTPS rather than HTTP.
For more help you can:
* Check the in-app documentation, in the top right corner of the app (it's also mirrored on [Stash-Docs](https://docs.stashapp.cc/in-app-manual))
* Join the [Matrix space](https://matrix.to/#/#stashapp:unredacted.org)
* Join the [Discord server](https://discord.gg/2TsNFKt), where the community can offer support.
* Start a [discussion on GitHub](https://github.com/stashapp/stash/discussions)
# Customization
## Themes and CSS Customization
There is a [directory of community-created themes](https://github.com/stashapp/stash/wiki/Themes) on our Wiki, along with instructions on how to install them.
There is a [directory of community-created themes](https://docs.stashapp.cc/user-interface-ui/themes) on Stash-Docs, along with instructions on how to install them.
You can also make Stash interface fit your desired style with [Custom CSS snippets](https://github.com/stashapp/stash/wiki/Custom-CSS-snippets) and [CSS Tweaks](https://github.com/stashapp/stash/wiki/CSS-Tweaks).
You can also change the Stash interface to fit your desired style with various snippets from [Custom CSS snippets](https://docs.stashapp.cc/user-interface-ui/custom-css-snippets).
# Support (FAQ)
# For Developers
Answers to other Frequently Asked Questions can be found [on our Wiki](https://github.com/stashapp/stash/wiki/FAQ)
Pull requests are welcome!
For issues not addressed there, there are a few options.
* Read the [Wiki](https://github.com/stashapp/stash/wiki)
* Check the in-app documentation (also available [here](https://github.com/stashapp/stash/tree/develop/ui/v2.5/src/docs/en)
* Join the [Discord server](https://discord.gg/2TsNFKt), where the community can offer support.
* Run `yarn install --frozen-lockfile` in the `stash/ui/v2.5` folder (before running make generate for first time).
NOTE: You may need to run the `go get` commands outside the project directory to avoid modifying the projects module file.
## Environment
### macOS
TODO
### Windows
1. Download and install [Go for Windows](https://golang.org/dl/)
2. Download and install [MingW](https://sourceforge.net/projects/mingw-w64/)
3. Search for "advanced system settings" and open the system properties dialog.
1. Click the `Environment Variables` button
2. Add `GO111MODULE=on`
3. Under system variables find the `Path`. Edit and add `C:\Program Files\mingw-w64\*\mingw64\bin` (replace * with the correct path).
NOTE: The `make` command in Windows will be `mingw32-make` with MingW.
## Commands
*`make generate` - Generate Go and UI GraphQL files
*`make build` - Builds the binary (make sure to build the UI as well... see below)
*`make pre-ui` - Installs the UI dependencies. Only needs to be run once before building the UI for the first time, or if the dependencies are updated
*`make fmt-ui` - Formats the UI source code.
*`make ui` - Builds the frontend and the packr2 files
*`make packr` - Generate packr2 files (sub-target of `ui`. Use to regenerate packr2 files without rebuilding UI)
*`make vet` - Run `go vet`
*`make lint` - Run the linter
*`make fmt` - Run `go fmt`
*`make fmt-check` - Ensure changed files are formatted correctly
*`make it` - Run the unit and integration tests
*`make validate` - Run all of the tests and checks required to submit a PR
*`make ui-start` - Runs the UI in development mode. Requires a running stash server to connect to. Stash port can be changed from the default of `9999` with environment variable `REACT_APP_PLATFORM_PORT`.
## Building a release
1. Run `make generate` to create generated files
2. Run `make ui` to compile the frontend
3. Run `make build` to build the executable for your current platform
## Cross compiling
This project uses a modification of the [CI-GoReleaser](https://github.com/bep/dockerfiles/tree/master/ci-goreleaser) docker container to create an environment
where the app can be cross-compiled. This process is kicked off by CI via the `scripts/cross-compile.sh` script. Run the following
command to open a bash shell to the container to poke around:
Stash can be profiled using the `--cpuprofile <output profile filename>` command line flag.
The resulting file can then be used with pprof as follows:
`go tool pprof <path to binary> <path to profile filename>`
With `graphviz` installed and in the path, a call graph can be generated with:
`go tool pprof -svg <path to binary> <path to profile filename> > <output svg file>`
See [Development](docs/DEVELOPMENT.md) and [Contributing](docs/CONTRIBUTING.md) for information on working with the codebase, getting a local development setup, and contributing changes.
This dockerfile is used to build a stash docker container using the current source code.
This dockerfile is used to build a stash docker container using the current source code. This is ideal for testing your current branch in docker. Note that it does not include python, so python-based scrapers will not work in this image. The production docker images distributed by the project contain python and the necessary packages.
# Building the docker container
From the top-level directory (should contain `main.go` file):
This dockerfile is used by travis to build the stash image. It must be run after cross-compiling - that is, `stash-linux` must exist in the `dist` directory. This image must be built from the `dist` directory.
This Dockerfile is used by CI to build the `stashapp/stash` Docker image. It must be run after cross-compiling - that is, `stash-linux` must exist in the `dist` directory. This image must be built from the `dist` directory.
Modified from https://github.com/bep/dockerfiles/tree/master/ci-goreleaser
When the dockerfile is changed, the version number should be incremented in the Makefile and the new version tag should be pushed to docker hub. The `scripts/cross-compile.sh` script should also be updated to use the new version number tag, and `.travis.yml` needs to be updated to pull the correct image tag.
When the Dockerfile is changed, the version number should be incremented in the Makefile and the new version tag should be pushed to Docker Hub. The GitHub workflow files also need to be updated to pull the correct image tag.
Installing StashApp can likely work on others if your OS either has it's own package manager or comes shipped with Docker and docker-compose.
# Docker Installation (for most 64-bit GNU/Linux systems)
StashApp is supported on most systems that support Docker and docker-compose. Your OS likely ships with or makes available the necessary packages.
## Dependencies
The goal is to avoid as many dependencies as possible so for now the only pre-requisites you are required to have are `curl`,`docker`, and `docker-compose`for the most part your understanding of the technologies can be superficial so long as you can follow commands and are open to reading a bit you should be fine.
Only`docker` and `docker-compose`are required. For the most part your understanding of the technologies can be superficial. So long as you can follow commands and are open to reading a bit, you should be fine.
### Docker
Docker is effectively the cross-platform software package repository it allows you to ship an entire environment in what's referred to as a container. Containers are intended to hold everything that is needed to ship what's required to run an application from one place to another with a degree of a standard that makes it easy for everyone along the way to reproduce the environment for their step in the chain.
The other side of docker is it brings everything that we would typically have to teach you about the individual components of your soon to be installed StashApp and ffmpeg, docker-compose wraps it up nicely in a handful of easy to follow steps that should result in the same environment on everyone's host.
The installation method we recommend is via the `docker.com` website however if your specific operating system's repository versions are at the latest along with docker you should be good to launch with you using whatever instructions you wish. The version of Docker we used in our deployment for testing this process was `Docker version 17.05.0-ce, build 89658be` however any versions later than this will be sufficient. At the writing of this tutorial, this was not the latest version of Docker.
#### Just the link to installation instructions, please
Instructions for installing on Ubuntu are at the link that follows:
If you plan on using other versions of OS you should at least aim to be a Linux base with an x86_64 CPU and the appropriate minimum version of the dependencies.
### Docker-compose
Docker Compose's role in this deployment is to get you a fully working instance of StashApp exactly as you would need it to have a reasonable instance for testing / developing on, you could technically deploy a live instance with this, but without a reverse proxy, is not recommended. You are encouraged to learn how to use the Docker-Compose format, but it's not a required prerequisite for getting this running you need to have it installed successfully.
Install Docker Compose via this guide below, and it is essential if you're using an older version of Linux to use the official documentation from Docker.com because you require the more recent version of docker-compose at least version 3.4 aka 1.22.0 or newer.
#### Just the link to installation instructions, please
https://docs.docker.com/compose/install/
### Install curl
This one's easy, copy paste.
```
apt update -y && \
apt install -f curl
```
Installation instructions are available below, and if your distrobution's repository ships a current version of docker, you may use that.
https://docs.docker.com/engine/install/
### Get the docker-compose.yml file
Now you can either navigate to the [docker-compose.yml](https://raw.githubusercontent.com/stashapp/stash/master/docker/production/docker-compose.yml) in the repository, OR you can make your Linux console do it for you with this.
Now you can either navigate to the [docker-compose.yml](https://raw.githubusercontent.com/stashapp/stash/develop/docker/production/docker-compose.yml) in the repository, or if you have curl, you can make your Linux console do it for you:
Once you have that file where you want it, you can either modify the settings as you please OR you can run the following to get it up and running instantly.
Once you have that file where you want it, modify the settings as you please, and then run:
```
cd ~ && docker-compose up -d
docker-compose up -d
```
Installing StashApp this way will by default bind stash to port 9999 or in web browser terms. http://YOURIP:9999 or if you're doing this on your machine locally which is the only recommended production version of this container as is with no security configurations set at all is http://localhost:9999
Installing StashApp this way will by default bind stash to port 9999. This is available in your web browser locally at http://localhost:9999 or on your network as http://YOUR-LOCAL-IP:9999
Good luck and have fun!
### Docker
Docker is effectively a cross-platform software package repository. It allows you to ship an entire environment in what's referred to as a container. Containers are intended to hold everything that is needed to run an application from one place to another, making it easy for everyone along the way to reproduce the environment.
The StashApp docker container ships with everything you need to automatically build and run stash, including ffmpeg.
### docker-compose
Docker Compose lets you specify how and where to run your containers, and to manage their environment. The docker-compose.yml file in this folder gets you a fully working instance of StashApp exactly as you would need it to have a reasonable instance for testing / developing on. If you are deploying a live instance for production, a reverse proxy (such as NGINX or Traefik) is recommended, but not required.
- an application for organising and viewing adult content - currently this is videos and images, in future this will be extended to include audio and text content
- organising includes scraping of metadata from websites and metadata repositories
- free and open-source
- portable and offline - can be run on a USB stick without needing to install dependencies (with the exception of ffmpeg)
- minimal, but highly extensible. The core feature set should be the minimum required to achieve the primary goal, while being extensible enough to extend via plugins
- easy to learn and use, with minimal technical knowledge required
The core stash system is not intended for:
- managing downloading of content
- managing content on external websites
- publically sharing content
Other requirements:
- support as many video and image formats as possible
- interfaces with external systems (for example stash-box) should be made as generic as possible.
Design considerations:
- features are easy to add and difficult to remove. Large superfluous features should be scrutinised and avoided where possible (eg DLNA, filename parser). Such features should be considered for third-party plugins instead.
## Technical Debt
Please be sure to consider how heavily your contribution impacts the maintainability of the project long term, sometimes less is more. We don't want to merge collossal pull requests with hundreds of dependencies by a driveby contributor.
1. Download and install [Go for Windows](https://golang.org/dl/)
2. Download and extract [MinGW64](https://sourceforge.net/projects/mingw-w64/files/) (scroll down and select x86_64-posix-seh, don't use the autoinstaller, it doesn't work)
3. Search for "Advanced System Settings" and open the System Properties dialog.
1. Click the `Environment Variables` button
2. Under System Variables find `Path`. Edit and add `C:\MinGW\bin` (replace with the correct path to where you extracted MingW64).
NOTE: The `make` command in Windows will be `mingw32-make` with MinGW. For example, `make pre-ui` will be `mingw32-make pre-ui`.
### macOS
1. If you don't have it already, install the [Homebrew package manager](https://brew.sh).
2. Install dependencies: `brew install go git yarn gcc make node ffmpeg`
### Linux
#### Arch Linux
1. Install dependencies: `sudo pacman -S go git yarn gcc make nodejs ffmpeg --needed`
- If you haven't already, [fetch the ports tree and verify](https://www.openbsd.org/faq/ports/ports.html#PortsFetch).
- Find the ffmpeg port in `/usr/ports/graphics/ffmpeg`, and patch the Makefile to include libwebp
- Add `webp` to `WANTLIB`
- Add `graphics/libwebp` to the list in `LIB_DEPENDS`
- Add `-lwebp -lwebpdecoder -lwebpdemux -lwebpmux` to `LIBavcodec_EXTRALIBS`
- Add `--enable-libweb` to the list in `CONFIGURE_ARGS`
- If you've already built ffmpeg from ports before, you may need to also increment `REVISION`
- Run `doas make install`
- Follow the instructions below to build a release, but replace the final step `make build-release` with `gmake flags-release stash`, to [avoid the PIE buildmode](https://github.com/golang/go/issues/59866).
NOTE: The `make` command in OpenBSD will be `gmake`. For example, `make pre-ui` will be `gmake pre-ui`.
## Commands
*`make pre-ui` - Installs the UI dependencies. This only needs to be run once after cloning the repository, or if the dependencies are updated.
*`make generate` - Generates Go and UI GraphQL files. Requires `make pre-ui` to have been run.
*`make generate-stash-box-client` - Generate Go files for the Stash-box client code.
*`make ui` - Builds the UI. Requires `make pre-ui` to have been run.
*`make stash` - Builds the `stash` binary (make sure to build the UI as well... see below)
*`make stash-macapp` - Builds the `Stash.app` macOS app (only works when on macOS, for cross-compilation see below)
*`make phasher` - Builds the `phasher` binary
*`make build` - Builds both the `stash` and `phasher` binaries, alias for `make stash phasher`
*`make build-release` - Builds release versions (debug information removed) of both the `stash` and `phasher` binaries, alias for `make flags-release flags-pie build`
*`make docker-build` - Locally builds and tags a complete 'stash/build' docker image
*`make docker-cuda-build` - Locally builds and tags a complete 'stash/cuda-build' docker image
*`make validate` - Runs all of the tests and checks required to submit a PR
*`make lint` - Runs `golangci-lint` on the backend
*`make it` - Runs all unit and integration tests
*`make fmt` - Formats the Go source code
*`make fmt-ui` - Formats the UI source code
*`make validate-ui` - Runs tests and checks for the UI only
*`make fmt-ui-quick` - (experimental) Formats only changed UI source code
*`make validate-ui-quick` - (experimental) Runs tests and checks of changed UI code
*`make server-start` - Runs a development stash server in the `.local` directory
*`make server-clean` - Removes the `.local` directory and all of its contents
*`make ui-start` - Runs the UI in development mode. Requires a running Stash server to connect to - the server URL can be changed from the default of `http://localhost:9999` using the environment variable `VITE_APP_PLATFORM_URL`, but keep in mind that authentication cannot be used since the session authorization cookie cannot be sent cross-origin. The UI runs on port `3000` or the next available port.
When building, you can optionally prepend `flags-*` targets to the target list in your `make` command to use different build flags:
*`flags-release` (e.g. `make flags-release stash`) - Remove debug information from the binary.
*`flags-pie` (e.g. `make flags-pie build`) - Build a PIE (Position Independent Executable) binary. This provides increased security, but it is unsupported on some systems (notably 32-bit ARM and OpenBSD).
*`flags-static` (e.g. `make flags-static phasher`) - Build a statically linked binary (the default is a dynamically linked binary).
*`flags-static-pie` (e.g. `make flags-static-pie stash`) - Build a statically linked PIE binary (using `flags-static` and `flags-pie` separately will not work).
*`flags-static-windows` (e.g. `make flags-static-windows build`) - Identical to `flags-static-pie`, but does not enable the `netgo` build tag, which is not needed for static builds on Windows.
## Local development quickstart
1. Run `make pre-ui` to install UI dependencies
2. Run `make generate` to create generated files
3. In one terminal, run `make server-start` to run the server code
4. In a separate terminal, run `make ui-start` to run the UI in development mode
5. Open the UI in a browser: `http://localhost:3000/`
Changes to the UI code can be seen by reloading the browser page.
Changes to the backend code require a server restart (`CTRL-C` in the server terminal, followed by `make server-start` again) to be seen.
On first launch:
1. On the "Stash Setup Wizard" screen, choose a directory with some files to test with
2. Press "Next" to use the default locations for the database and generated content
3. Press the "Confirm" and "Finish" buttons to get into the UI
4. On the side menu, navigate to "Tasks -> Library -> Scan" and press the "Scan" button
5. You're all set! Set any other configurations you'd like and test your code changes.
To start fresh with new configuration:
1. Stop the server (`CTRL-C` in the server terminal)
2. Run `make server-clean` to clear all config, database, and generated files (under `.local`)
3. Run `make server-start` to restart the server
4. Follow the "On first launch" steps above
## Building a release
Simply run `make` or `make release`, or equivalently:
1. Run `make pre-ui` to install UI dependencies
2. Run `make generate` to create generated files
3. Run `make ui` to build the frontend
4. Run `make build-release` to build a release executable for your current platform
## Cross-compiling
This project uses a modification of the [CI-GoReleaser](https://github.com/bep/dockerfiles/tree/master/ci-goreleaser) Docker container for cross-compilation, defined in `docker/compiler/Dockerfile`.
To cross-compile the app yourself:
1. Run `make pre-ui`, `make generate` and `make ui` outside the container, to generate files and build the UI.
2. Pull the latest compiler image from Docker Hub: `docker pull stashapp/compiler`
3. Run `docker run --rm --mount type=bind,source="$(pwd)",target=/stash -w /stash -it stashapp/compiler /bin/bash` to open a shell inside the container.
4. From inside the container, run `make build-cc-all` to build for all platforms, or run `make build-cc-{platform}` to build for a specific platform (have a look at the `Makefile` for the list of targets).
5. You will find the compiled binaries in `dist/`.
NOTE: Since the container is run as UID 0 (root), the resulting binaries (and the `dist/` folder itself, if it had to be created) will be owned by root.
## Profiling
Stash can be profiled using the `--cpuprofile <output profile filename>` command line flag.
The resulting file can then be used with pprof as follows:
`go tool pprof <path to binary> <path to profile filename>`
With `graphviz` installed and in the path, a call graph can be generated with:
`go tool pprof -svg <path to binary> <path to profile filename> > <output svg file>`
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.