* Add counter to File Info where file count > 1
* Add file modification time to File Info panel
* Remove duplicate intl keys
* Add file count to duplicate checker
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Treat no output from ffmpeg as an error condition
* Distinguish file vs. video duration, and use later where appropriate
* Check for empty file in generateFile
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* track watchtime and view time
* add view count sorting, added continue position filter
* display metrics in file info
* add toggle for tracking activity
* save activity every 10 seconds
* reset resume when video is nearly complete
* start from beginning when playing scene in queue
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make read-only operations use WithReadTxn
* Allow one database write thread
* Add unit test for concurrent transactions
* Perform some actions after commit to release txn
* Suppress some errors from cancelled context
* graphql: support date and timestamp filter types
* sql: add support for date & timestamp criterions
* ui: add support for date and timestamp criterions
* scenes: add support for filtering by date, created at and updated at
* image: support filtering by created at and updated at
* gallery: support filtering by date, created at and updated at
* movie: support filtering by date, created at and updated at
* studio: support filtering by date, created at and updated at
* tag: support filtering by date, created at and updated at
* performer: support filtering by bitrh & death date and created & updated at
* marker: support filtering by created & updated at and scene date, created & updated at
* Reassign scene file functionality
* Implement scene create
* Add scene create UI
* Add sceneMerge backend support
* Add merge scene to UI
* Populate split create with scene details
* Add merge button to duplicate checker
* Handle file-less scenes in marker preview generate
* Make unique file name for file-less scene exports
* Add o-counter to scene update input
* Hide rescan for file-less scenes
* Generate heatmap if no speed set on file
* Fix count in scene/image queries
Introducing a limit to how many options are shown in select dropdowns. Fixes an issue I was experiencing where large numbers of options (5000 tags) was causing dropdown to be unresponsive. Does not effect filtering, always shows 'Create "..."' option if it exists, and shows a notice at the bottom of the dropdown of how many options were hidden from the list if any were.
* added schema migration and updated data models
* added code and director to UI
* new fields are exported and imported
* added filters
* Add changelog entry
* Add types to player plugins
* Use videojs-vtt.js to parse sprite VTT files
* Overhaul scene player
* Replace vtt-thumbnails-freetube
* Remove chapters_vtt
* Force remove shadow from player progress bar
* Cleanup player css
* Rewrite live.ts as middleware
* Don't force play when changing source
* Fire handlers when file updated or moved
* Create galleries as needed
* Clean empty galleries
* Handle cleaning zip folders when path changed
* Fix gallery association on duplicate images
* Re-create missing folder-based galleries
* Only update fingerprints if changed
* Fix panic when loading primary file fails
* Fix gallery/scene association
* Fix display of scene gallery in card
* Use natural_cs collation with paths for title sorting
* Don't recalculate MD5 if not enabled
Remove MD5 if oshash has changed and MD5 was not calculated.
* Fix panic in paged DLNA
* Prevent identical hashes in stash-box drafts
* Fix incorrect timestamp updates
* Correct folder time fields
* Add migration with new indexes
* Correct mod_time format
* Add mod_time to data massage
* Load scene relationships on demand
* Load image relationships on demand
* Load gallery relationships on demand
* Add dataloaden
* Use dataloaders
* Use where in for other find many functions
* Do database txn in same thread. Retry on locked db
* Remove captions from slimscenedata
* Fix tracing
* Use where in instead of individual selects
* Remove scenes_query view
* Remove image query view
* Remove gallery query view
* Use where in for FindMany
* Don't interrupt scanning zip files
* Fix image filesize sort
* Use cache during migration
* Avoid use of query views
* Use FindMany to find related objects
* Log slow queries
* Add folders to generated files
* Use SlimScene for scene queries
* Include filename in migration error message
* Fix destroy gallery not destroying file
* Re-add minModTime functionality
* Deprecate useFileMetadata and stripFileExtension
* Optimise files post migration
* Decorate moved files. Use first missing file in move
* Include path in thumbnail generation error log
* Fix stash-box draft submission
* Don't destroy files unless deleting
* Call handler for files with no associated objects
* Fix moved zips causing error on scan
* Restructure data layer part 2 (#2599)
* Refactor and separate image model
* Refactor image query builder
* Handle relationships in image query builder
* Remove relationship management methods
* Refactor gallery model/query builder
* Add scenes to gallery model
* Convert scene model
* Refactor scene models
* Remove unused methods
* Add unit tests for gallery
* Add image tests
* Add scene tests
* Convert unnecessary scene value pointers to values
* Convert unnecessary pointer values to values
* Refactor scene partial
* Add scene partial tests
* Refactor ImagePartial
* Add image partial tests
* Refactor gallery partial update
* Add partial gallery update tests
* Use zero/null package for null values
* Add files and scan system
* Add sqlite implementation for files/folders
* Add unit tests for files/folders
* Image refactors
* Update image data layer
* Refactor gallery model and creation
* Refactor scene model
* Refactor scenes
* Don't set title from filename
* Allow galleries to freely add/remove images
* Add multiple scene file support to graphql and UI
* Add multiple file support for images in graphql/UI
* Add multiple file for galleries in graphql/UI
* Remove use of some deprecated fields
* Remove scene path usage
* Remove gallery path usage
* Remove path from image
* Move funscript to video file
* Refactor caption detection
* Migrate existing data
* Add post commit/rollback hook system
* Lint. Comment out import/export tests
* Add WithDatabase read only wrapper
* Prepend tasks to list
* Add 32 pre-migration
* Add warnings in release and migration notes
* Use primitive string in recommendation row props
* Use unique keys in recommendation rows
The keys for the cards used while loading clash with the ids of the actual cards, causing a list unique key warning.
* List filter alignment tweaks
* Rework list hook filtering
* Internationalise checksum correctly
* Use hotkeys '[' and ']' to scrub video player forwards and backwards by 10% of the scene
* Don't loop back to beginning
* Add manual keybind entry
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* refactored common code in recommendation row
* Implement front page options in config
* Allow customisation from front page
* Rename recommendations to front page
* Add generic UI settings
* Support adding premade filters
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
The incorrect error check means that images is always 0, which means scanImages is also always true, which causes the images inside the zip to be unnecessarily rescanned every time.
* Changed Most Active Studios to Latest Studios
* dynamically create view all link and created message for view all
* created shared determineSlidesToScroll method
* removed added code in Shared/index.ts
* renamed getSlickSettings to getSlickSliderSettings
* Updated row headers to follow plex naming convention
* removed extra s in Sceness
* updated row header css to better align header text with view all anchor
* Refactor interactive into context
* Stop the interactive device when leaving page
* Show interactive state if not ready
* Handle navigation and looping
* Rename manager.instance to Manager
* Show dialog message on fatal error on Windows
* Hide console windows explicitly on icon launch
Gets rid of the windowsgui flag, which causes all sorts of issues.
Instead, checks if stash was launched from the icon, and if so hides the console.
* Remove fixconsole
* Add changelog entries
* Make the script scraper context-aware
Connect the context to the command execution. This means command
execution can be aborted if the context is canceled. The context is
usually bound to user-interaction, i.e., a scraper operation issued
by the user. Hence, it seems correct to abort a command if the user
aborts.
* Enable errchkjson
Some json marshal calls are *safe* in that they can never fail. This is
conditional on the types of the the data being encoded. errchkjson finds
those calls which are unsafe, and also not checked for errors.
Add logging warnings to the place where unsafe encodings might happen.
This can help uncover usage bugs early in stash if they are tripped,
making debugging easier.
While here, keep the checker enabled in the linter to capture future
uses of json marshalling.
* Pass the context for zip file scanning.
* Pass the context in scanning
* Pass context, replace context.TODO()
Where applicable, pass the context down toward the lower functions in
the call stack. Replace uses of context.TODO() with the passed context.
This makes the code more context-aware, and you can rely on aborting
contexts to clean up subsystems to a far greater extent now.
I've left the cases where there is a context in a struct. My gut feeling
is that they have solutions that are nice, but they require more deep
thinking to unveil how to handle it.
* Remove context from task-structs
As a rule, contexts are better passed explicitly to functions than they
are passed implicitly via structs. In the case of tasks, we already
have a valid context in scope when creating the struct, so remove ctx
from the struct and use the scoped context instead.
With this change it is clear that the scanning functions are under a
context, and the task-starting caller has jurisdiction over the context
and its lifetime. A reader of the code don't have to figure out where
the context are coming from anymore.
While here, connect context.TODO() to the newly scoped context in most
of the scan code.
* Remove context from autotag struct too
* Make more context-passing explicit
In all of these cases, there is an applicable context which is close
in the call-tree. Hook up to this context.
* Simplify context passing in manager
The managers context handling generally wants to use an outer context
if applicable. However, the code doesn't pass it explicitly, but stores
it in a struct. Pull out the context from the struct and use it to
explicitly pass it.
At a later point in time, we probably want to handle this by handing
over the job to a different (program-lifetime) context for background
jobs, but this will do for a start.
* Upgrade gqlgen to v0.17.2
This enables builds on Go 1.18. github.com/vektah/gqlparser is upgraded
to the newest version too.
Getting this to work is a bit of a hazzle. I had to first remove
vendoring from the repository, perform the upgrade and then re-introduce
the vendor directory. I think gqlgens analysis went wrong for some
reason on the upgrade. It would seem a clean-room installation fixed it.
* Bump project to 1.18
* Update all packages, address gqlgenc breaking changes
* Let `go mod tidy` handle the go.mod file
* Upgrade linter to 1.45.2
* Introduce v1.45.2 of the linter
The linter now correctly warns on `strings.Title` because it isn't
unicode-aware. Fix this by using the suggested fix from x/text/cases
to produce unicode-aware strings.
The mapping isn't entirely 1-1 as this new approach has a larger iface:
it spans all of unicode rather than just ASCII. It coincides for ASCII
however, so things should be largely the same.
* Ready ourselves for errchkjson and contextcheck.
* Revert dockerfile golang version changes for now
Co-authored-by: Kermie <kermie@isinthe.house>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Replace JW Player with video.js
* Move HLS stream to bottom of list
HLS doesn't work very well on non-ios devices.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Move main to cmd
* Move api to internal
* Move logger and manager to internal
* Move shell hiding code to separate package
* Decouple job from desktop and utils
* Decouple session from config
* Move static into internal
* Decouple config from dlna
* Move desktop to internal
* Move dlna to internal
* Decouple remaining packages from config
* Move config into internal
* Move jsonschema and paths to models
* Make ffmpeg functions private
* Move file utility methods into fsutil package
* Move symwalk into fsutil
* Move single-use util functions into client package
* Move slice functions to separate packages
* Add env var to suppress windowsgui arg
* Move hash functions into separate package
* Move identify to internal
* Move autotag to internal
* Touch UI when generating backend
* Continue identify if source fails
* Handle empty result set correctly
* Parse null values from scraper script correctly
* Omit warning when json selector value missing
* Return nil when scraped item not found
* Fix graphql validation errors
* Add duration to autotag finish message
* No sorting scene/image/gallery where not specified
* Use an LRU cache for sqlite regexp function
* Compile path separator regex once
* Cache objects with single letter first names
* Move finished auto-tag log
* Add more verbose logging
* Add new changelog
* Remove single unicode character from autotag query
* Compile regex once where possible
* Fix CPU profiling
* Only match unicode characters if in path
* Performer tagger modal will load country name instead of code
When using the performer tagger, it would pull the 2 character country code from StashDB and store that. It now shows the full country name and stores the full country name instead. Also verified that it causes no issue with a performer that has no country set on StashDB
* Make changes to fix special characters in Criterion labels (#1819)
Reverse the '&' and '+' replacement done on StringCriterion
Decodes special characters in IHierarchicalLabeledIdCriterion
* Have Gallery info panel say "Downloaded From" next to URL
In the gallery info panel it says "Path" next to the URL. This updates it to say "Downloaded From" next to the URL, like it does in the scene view for the URL.
* Add country to EditPerformersDialog
* Add most text fields to EditPerformersDialog
* Refactor to pass validate
* Remove height and measurements fields
* Add gender field
* Run fmt-ui
* Cleaned up some language.
- Optimized the Quick Start guide, moved "how do i get to stash" from FAQ, moved some sections together.
* as above
* Update README.md
Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
* Some editing and notes
* Remove double click line, let's not patronize :)
Co-authored-by: kermieisinthehouse <kermie@isinthe.house>
Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
* Exposed created_at and updated_at dates on the detail panels for images and scenes
* Add fields to gallery page
* Internationalisation
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add InteractiveSpeed to scene model
* add InteractiveHeatmapSpeedGenerator
* add GenerateInteractiveHeatmapSpeedTask
* add InteractiveHeatmapSpeedTask to GenerateJob
* add InteractiveHeatmap on sceneRoutes
* delete heatmap when scene is destroyed
* render interactive heatmap in GridCard
* render InteractiveSpeed on SceneCard
* render InteractiveSpeed in SceneFileInfoPanel
* InteractiveSpeed filters
* Added joinType to join struct
* Added addInnerJoin function to perform INNER JOIN type of joins
* Added innerJoin function to perform INNER JOIN type of joins
* Use inner joins when querying images in a gallery
* Renamed addJoin to addLeftJoin
* Move scan options out of dialog
* Move autotag and clean options out of dialogs
* Move generate options out of dialog
* Animate button while tasks running
* Revert to earlier Tasks UI iteration
* Rearrange and clarify scan options
* Support a maxAge input on metadata scans.
Extend the GraphQL world with a Duration scalar. It is parsed as a
typical Go duration, i.e., "4h" is 4 hours. Alternatively, one can
pass an integer which is interpreted as seconds.
Extend Mutation.metadataScan(input: $input) to support a new optional
value, maxAge. If set, the scanner will exit early if the file it
is looking at has an mtime older than the cutOff point generated by
now() - maxAge
This speeds up scanning in the case where the user knows how old the
changes on disk are, by exiting the scan early if that is the case.
* Change maxAge into minModTime
Introduce a `Timestamp` scalar, so we have a scalar we control. Let
it accept three formats:
* RFC3339Nano
* @UNIX where UNIX is a unix-timestamp: seconds after 01-01-1970
* '<4h': a timestamp relative to the current server time
This scalar parses to a time.Time.
Use MinModTime in the scanner to filter out a large number of scan
analyzes by exiting the scan operation early.
* Heed the linter, perform errcheck
* Rename test vars for consistency.
* Code review: move minModTime into queuefiles
* Remove the ability to input Unix timestamps
Test failures on the CI-system explains why this is undesirable. It is
not clear what timezone one is operating in when entering a unix
timestamp. We could go with UTC, but it is so much easier to require an
RFC3339 timestamp, which avoids this problem entirely.
* Move the minModTime field into filters
Create a new filter input object for metadata scans, and push the
minModTime field in there. If we come up with new filters, they can
be added to that input object rather than cluttering the main input
object.
* Use utils.ParseDateStringAsTime
Replace time.Parse with utils.ParseDateStringAsTime
While here, add some more test cases for that parser.
* Push scrapeByURL into scrapers
Replace ScrapePerfomerByURL, ScrapeMovie..., ... with ScrapeByURL in
the scraperActionImpl interface. This allows us to delete a lot of
repeated code in the scrapers and replace the central part with a
switch on the scraper type.
* Fold name scraping into one call
Follow up on scraper refactoring. Name scrapers use the same code path.
This allows us to restructure some code and kill some functions, adding
variance to the name scraping code. It allows us to remove some code
repetition as well.
* Do not export loop refs.
* Simplify fragment scraping
Generalize fragment scrapers into ScrapeByFragment. This simplifies
fragment code flows into a simpler pathing which should be easier
to handle in the future.
* Eliminate more context.TODO()
In a number of cases, we have a context now. Use the context rather than
TODO() for those cases in order to make those operations cancellable.
* Pass the context for the stashbox scraper
This removes all context.TODO() in the path of the stashbox scraper,
and replaces it with the context that's present on each of the paths.
* Pass the context into subscrapers
Mostly a mechanical update, where we pass in the context for
subscraping. This removes the final context.TODO() in the scraper
code.
* Warn on unknown fields from scripts
A common mistake for new script writers are that they return fields
not known to stash. For instance the name "description" is used rather
than "details".
Decode disallowing unknown fields. If this fails, use a tee-reader to
fall back to the old behavior, but print a warning for the user in this
case. Thus, we retain the old behavior, but print warnings for scripts
which fails the more strict unknown-fields detection.
* Nil-check before running the postprocessing chain
Fixes panics when scraping returns nil values.
* Lift nil-ness in post-postprocessing
If the struct we are trying to post-process is nil, we shouldn't
enter the postprocessing flow at all. Pass the struct as a value
rather than a pointer, eliminating nil-checks as we go. Use the
top-level postProcess call to make the nil-check and then abort there
if the object we are looking at is nil.
* Allow conversion routines to handle values
If we have a non-pointer type in the interface, we should also convert
those into ScrapedContent. Otherwise we get errors on deprecated
functions.
* Add scan dialog
* Add Auto Tag dialog
* Refactor and combine Generate dialog
* Add clean dialog
* Support scan task default setting
* Support saving auto tag defaults
* Support for generate defaults
* Simplify scraper listing
Introduce an enum, scraper.Kind, which explains what we are looking
for. Make it possible to match this from a scraper struct.
Use the enum to rewrite all the listing code to use the same code path.
* Use a map, nitpick ScrapePerformerList
Let the cache store a map from ID of a scraper to the scraper. This
improves lookups when there are many scrapers, making it practically
O(1) rather than O(n). If many scrapers are stored, this is faster.
Since range expressions work unchanged, we don't have to change much,
and things will still work.
make Kind a Stringer
Rename ScraperPerformerList -> ScraperPerformerQuery since that name
is used in the other scrapers, and we value consistency.
Tune ScraperPerformerQuery:
* Return static errors
* Use the new functionality
* When loading scrapers, do so directly
Rather than first walking the directory structure to obtain file paths,
fold the load directly in the the filepath walk. This makes the code
for more direct.
* Use static ErrNotFound
If a scraper isn't found, return one static error. This paves the way
for eventually doing our own error-presenter in gqlgen.
* Store the cache in the Resolver state
Putting the scraperCache directly in the resolver avoids the need to
call manager.GetInstance() all over the place to get access to the
scraper cache. The cache is stored by pointer, so it should be safe,
since the cache will just update its internal state rather than being
overwritten.
We can now utilize the resolver state to grab the cache where needed.
While here, pass context.Context from the resolver down into a function,
which removes a context.TODO()
* Introduce ScrapedContent
Create a union in the GraphQL schema for all scraped content. This
simplifies the internal implementation because we get variance on
the output content type.
Introduce a new type ScrapedContentType which signifies the scraped
content you want as a caller.
Use these to generalize the List interface and the URL scraping
interface.
* Simplify the scraper API
Introduce a new interface for scraping. This interface is then
used in the upper half of the scraper code, to make the code use one
code flow rather than multiple code flows. Variance is currently at
the old scraper structure.
Add extending interfaces for the different ways of invoking scrapes.
Use interface conversions to convert a scraper from the cache to a
scraper supporting the extra methods.
The return path returns models.ScrapedContent.
Write a general postProcess function in the scraper, handling all
ScrapedContent via type switching. This consolidates all postprocessing
code flows.
Introduce marhsallers in the resolver code for converting ScrapedContent
into the underlying concrete types. Use this to plug the existing
fields in the Query resolver, so everything still works.
* ScrapedContent: add more marshalling functions
Handle all marshalling of ScrapedContent through marhsalling functions.
Removes some hand-rolled early variants of it, and replaces it with
a canonical code flow.
* Support loadByName via scraper_s
In order to temporarily plug a hole in the current implementation, we
use the older implementation as a hook to get the newer implementation
to run.
Later on, this can serve as a guide for how to implement the lower level
bits inside the scrapers themselves. For now, it just enables support.
* Plug the remaining scraper functions for now
Since we would like to have a scraper which works in between refactors,
plug the lower level parts of the scraper for now. It avoids us having
to tackle this part just yet.
* Move postprocessing to its own file
There's enough postprocessing to clutter the main scrapers.go file.
Move all of this into a new file, postprocessing to make the API
simpler. It now lives in scrapers.go.
* Scraper: Invoke API consistency
scraper.Cache.ScrapeByName -> ScrapeName
* Fix scraping scenes by URL
Simple typo. While here, also make a single marshaller nil-aware.
* Introduce scraper groups, consolidate loadByURL
Rename `scraper_s` into `group`. A group is a group of scrapers with
the same identity. This corresponds to a single YAML file for a scraper
configuration. It defines a group which supports different types of
scraping contexts.
Move config into the group, and lift txnManager and globalConfig to
the group.
Because we now return models.ScrapedContent we can use interfaces to
get variance from the different underlying scrapers. Use a type
switch for the URL matcher candidates. And then again for the scrapers.
This consolidates all URL scraping paths into one.
While here, remove the urlMatcher interface which isn't needed. Also
clean up the remaining interfaces for url scraping and delete code
which has no purpose anymore.
* Consolidate fragment scraping in one code path
While here, abide the linters checks.
* Refactor loadByFragment
Give it the same treatment as loadByURL:
Step 1: find a scraperActionImpl which works for the data.
Step 2: use that to scrape
Most of this is simple analysis on the data at hand. It can be pushed
down further in a later commit, but for now we leave it here.
* Remove configScraper, autotag is a scraper
Remove the remains of the configScraper struct. It now lives on in the
group struct. Kill the remaining interfaces from the old implementation
while here.
Remove group.specification since it can now be handled by a simple
func call to spec().
Work through the autotag scraper. It now implements the scraper
interface, so it can be used as a scraper. This also simplifies the
autotag scraper quite a bit since it doens't have to implement a number
of unsupported func calls.
* Simplify the fragment scraper flow
* Pass the context
Eliminate a round of context.TODO() in the scraper code by passing
the calling context down into the subsystem. This will gracefully
allow for termination of remote calls if the client goes away for some
reason in GraphQL requests.
* Improve listScrapers in the schema
Support lists of types we accept.
* Be graceful on nil values in conversion
Supporting nil-values make the API more robust in the
case of partial results in a multi-scrape situation.
* Improve listScrapers: output at-most-once
Use the ID of a scraper to reduce the output set. If a scraper has
been included, don't include it again.
* Consolidate all API level errors into resolver.go
* Reorder files and functions:
scrapers.go -> cache.go:
It almost contains nothing but the cache code.
Move errors into scraper.go from here because
It is a better place to have them living right now
group.go:
All of the group structure. This can now go from
scraper.go, making it more lean. Move group create
from config_scraper to here.
config.go:
Move the `(c config) spec()` call to here.
config_scraper.go:
Empty file by now
* Name-update the scraper interfaces
Use 'via' rather than 'loadBy'.
The scrape happens via a given scrape method, so I think this is a nice
name for it.
* Rename scrapers for consistency.
While here, improve the error formatting, so different errors come
back differently.
* Nuke the freeones field from the GraphQL schema
* Fix autotag interfacing, refactor
The autotag scraper uses a pointer receiver, but the rest of the code
we use for scraping doesn't expect a pointer-receiver. Hence, to fix
the autotag scraper, we change it to be a value receiver, like the
rest of the code.
Fix: viaScene, and viaGallery.
While here, remove a couple of pointer-receiver methods which can be
trivially rewritten into plain functions.
* Protect against pointer interfaces
The underlying code can be a bit inconsistent in what it returns.
Introduce pointer-types in the postprocessing layer and handle them
accordingly for now. Once a better understanding of the lower levels
are understood, we can lift this.
* Move ErrConversion into the models package.
The conversion error pertains to the logic of converting models.
Because of this, it should move there, so it is centralized.
* Be consistent in scraper resolver error handling
If we have a static error
Err = errors.New(..)
Then use it wrapped at the start:
fmt.Errorf("%w: ...context...", Err)
This reads better.
While here, avoid using the underlying Atoi errors: they are verbose,
and like 99% of the time, the user know what is wrong from the input
string, so just give that back.
Also, remove the scraper id from the error contexts: it is implicit,
and the error wouldn't change if we used a different scraper, which
the error message would imply.
* Mark the list*Scrapers() API as deprecated
The same functionality is now present in listScrapers.
* Improve error formatting
Think about how each error is going to be used and tweak them to be
nicer.
* Return a sorted list of scrapers
This helps testing, it's closer to what we had, caches like stable data,
and it is easier for humans. It also makes the output stable, because
map iteration is randomized.
* Fix listScrapers calls to return in ID-order
Since we need the ordering to be by ID in all situations, it is easier
to just generalize the cache listScrapers call to support multiple
scraper types.
This avoids a de-dupe map up the chain, since every scraper is only
considered once. Sorting now happens in the cache listScrapers call.
Use this generalized function in all resolvers, which are now simple
passthroughs.
* Remove UpdateConfig from the scraper cache.
This isn't needed, so get rid of it.
* Pull a context into identify
Scraping scenes in the identify tasks now use a context from up the
call chain.
* Do not store the scraper cache in the resolver.
Scraper caches are updated through
manager.singleton•RefreshScraperCache, so we can't keep a pointer to
it in the resolver. Instead, solve this by adding a fetcher method to
the resolver type. This keeps it local to the resolver, while handling
the problem of updating caches in the configuration.
* Fix incorrect tense in toast
* Rename create_entity to emphasize past tense
* Localize "Started XXX" toasts
* Localize new zh-tw texts
* Refactor "continue" into the "actions" group
* Separate overrides from config
* Don't allow changing overridden value
* Write default host and port to config file
* Use existing library value. Hide generated if set
* Support Is (not) null for all multi criterions
Add support for the Is null and Is not null modifiers for all cases of
the MultiCriterionInput and HierarchicalMultiCriterionInput. This
partially overlaps the "X Count" filter which sometimes is available
(because it would be the same as "X Count equals 0" and "X Count greater
than 0") but this also enables it for other criterions like the "Parent
Studio" filter for studios or just the "Studios" filter for scenes /
images / galleries, the "Movies" filter for scenes etc.
* Don't crash UI on bad saved filter
* Add missing code for tag parent/child
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* update tag hierarchy validation
* refactor MergeHierarchy
* update tag hierarchy error message
* rename tag hierarchy function
* add tag path to error message
* Rename EnsureHierarchy to ValidateHierarchy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix responsive layout
* Refactor MainNavbar
* Stick the navbar to the bottom on mobile
* Fix menu item icon-text vertical alignment
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add delete file and generated files by default config options
* add alert message with files to be deleted
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Split performerCreate page into separate page
* Split studioCreate into a separate page
* Remove Partial types from performer/studio
* Split tagCreate into a separate page
* Split movieCreate into a separate page
* Split out galleryCreate into its own page
* Add loader to scene page
* Fix performer name fallback
* Fix movie layout shift
* Fix prompt comment and switch studio prompt to localized string
The version checking code performs its own error management and will
not pass errors to the caller. Hence, it needs to be aware of the types
of errors which can be returned.
In particular, the context.Canceled error will be returned if the
context is aborted through cancelation. This happens when the request
is terminated by tapping CTRL-C or if the browser request is terminated
while we are sitting waiting for the GH API.
* Docker CI builds: half the size, less than half the build time
* Add an "Official Build" Designator
* Fix .git constantly invalidating build cache, use distro ffmpeg
* Fix official build detection, add some compiler image docs
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make copies of buffers
Avoid reusing one of the incoming arrays as a append extension, and
make a copy of the data. It's cleaner in the long run and possibly
easier for the GC to maintain.
* Avoid appendAssign problems in tag code
Reuse the existing slice when appending.
* Fix appendAssign in encoder_scene_preview_chunk
Appending and creating a new slice is somewhat unintuitive since the
underlying slice might be extended to satisfy the new capacity. This
sometimes leads to faulty logic.
Rewrite the code so it reuses `args` for all appending, and builds one
array clearly in the code. It follows the general style of the function
where `args` is being built in small incremental batches and avoids
the introduction of new names.
* Enable the appendAssign check
This makes us pass all gocritic warnings.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add Cookies directly to the request
Rather than maintaining a cookie jar on a one-shot HTTP client, maintain
the jar ourselves: make a new jar, then use it to select the right
cookies.
The cookies are set on the request rather than on the client. This will
retain the current behavior as we are always throwing the client away
after each use.
This patch enables the lifting of the http client as well over time.
* Introduce a cached scraper HTTP client
The scraper cache is augmented with an *http.Client. These are safe for
concurrent use, so the pointer can safely be passed around. Push this
into scraper configurations where applicable, next to the txnManagers.
When we issue a loadUrl request, do so on the cached *http.Client,
which will reuse existing idle connections in the client if any are
present.
* Set MaxIdleConnsPerHost. Closes#1850
We allow for up to 8 idle connections to a single host. This should
make concurrent operation toward the same host reuse connections, even
for sizeable concurrency.
The number isn't bumped excessively high. We should probably limit
concurrency toward a single site anyway, since we'll be able to overrun
a site with queries quite easily if we have many concurrent goroutines
issuing requests at the same time.
* Reinstate driverOptions / useCDP check
Use DeMorgan's laws to invert the logic and exit early. Fixes tests
breaking.
* Documentation fixup.
* Use the scraper http.Client when fetching images
Fold image fetchers onto the cached scraper http.Client as well. This
makes the scraper have a single http.Client cache for all its
operations.
Thread the client upwards to the relevant attachment points: either the
cache, or a stash_box instance, which is extended to include a pointer
to the client.
Style roughly follows that of txnManagers.
* Use the same http Client as the GraphQL client use
Rather than using http.DefaultClient, use the same client as the
GraphQL client use in the stash_box subsystem. This localizes the
client used in the subsystem into the constructing New.. call.
* Hoist HTTP client construction
Create a function for initializaing the HTTP Client we use. While here
hoist magic numbers into constants. Introduce a proper static redirect
error and use it in the client code as well.
* Reinstate printCookies
This is a debugging function, and it might still come in handy in the
future at some point.
* Nitpick comment.
* Minor tidy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add a space after // comments
For consistency, the commentFormatting lint checker suggests a space
after each // comment block. This commit handles all the spots in
the code where that is needed.
* Rewrite documentation on functions
Use the Go idiom of commenting:
* First sentence declares the purpose.
* First word is the name being declared
The reason this style is preferred is such that grep is able to find
names the user might be interested in. Consider e.g.,
go doc -all pkg/ffmpeg | grep -i transcode
in which case a match will tell you the name of the function you are
interested in.
* Remove old code comment-blocks
There are some commented out old code blocks in the code base. These are
either 3 years old, or 2 years old. By now, I don't think their use is
going to come back any time soon, and Git will track old pieces of
deleted code anyway.
Opt for deletion.
* Reorder imports
Split stdlib imports from non-stdlib imports in files we are touching.
* Use a range over an iteration variable
Probably more go-idiomatic, and the code needed comment-fixing anyway.
* Use time.After rather than rolling our own
The idiom here is common enough that the stdlib contains a function for
it. Use the stdlib function over our own variant.
* Enable the commentFormatting linter
* Don't capitalize local variables
ValidCodecs -> validCodecs
* Capitalize deprecation markers
A deprecated marker should be capitalized.
* Use re.MustCompile for static regexes
If the regex fails to compile, it's a programmer error, and should be
treated as such. The regex is entirely static.
* Simplify else-if constructions
Rewrite
else { if cond {}}
to
else if cond {}
* Use a switch statement to analyze formats
Break an if-else chain. While here, simplify code flow.
Also introduce a proper static error for unsupported image formats,
paving the way for being able to check against the error.
* Rewrite ifElse chains into switch statements
The "Effective Go" https://golang.org/doc/effective_go#switch document
mentions it is more idiomatic to write if-else chains as switches when
it is possible.
Find all the plain rewrite occurrences in the code base and rewrite.
In some cases, the if-else chains are replaced by a switch scrutinizer.
That is, the code sequence
if x == 1 {
..
} else if x == 2 {
..
} else if x == 3 {
...
}
can be rewritten into
switch x {
case 1:
..
case 2:
..
case 3:
..
}
which is clearer for the compiler: it can decide if the switch is
better served by a jump-table then a branch-chain.
* Rewrite switches, introduce static errors
Introduce two new static errors:
* `ErrNotImplmented`
* `ErrNotSupported`
And use these rather than forming new generative errors whenever the
code is called. Code can now test on the errors (since they are static
and the pointers to them wont change).
Also rewrite ifElse chains into switches in this part of the code base.
* Introduce a StashBoxError in configuration
Since all stashbox errors are the same, treat them as such in the code
base. While here, rewrite an ifElse chain.
In the future, it might be beneifical to refactor configuration errors
into one error which can handle missing fields, which context the error
occurs in and so on. But for now, try to get an overview of the error
categories by hoisting them into static errors.
* Get rid of an else-block in transaction handling
If we succesfully `recover()`, we then always `panic()`. This means the
rest of the code is not reachable, so we can avoid having an else-block
here.
It also solves an ifElse-chain style check in the code base.
* Use strings.ReplaceAll
Rewrite
strings.Replace(s, o, n, -1)
into
strings.ReplaceAll(s, o, n)
To make it consistent and clear that we are doing an all-replace in the
string rather than replacing parts of it. It's more of a nitpick since
there are no implementation differences: the stdlib implementation is
just to supply -1.
* Rewrite via gocritic's assignOp
Statements of the form
x = x + e
is rewritten into
x += e
where applicable.
* Formatting
* Review comments handled
Stash-box is a proper noun.
Rewrite a switch into an if-chain which returns on the first error
encountered.
* Use context.TODO() over context.Background()
Patch in the same vein as everything else: use the TODO() marker so we
can search for it later and link it into the context tree/tentacle once
it reaches down to this level in the code base.
* Tell the linter to ignore a section in manager_tasks.go
The section is less readable, so mark it with a nolint for now. Because
the rewrite enables a ifElseChain, also mark that as nolint for now.
* Use strings.ReplaceAll over strings.Replace
* Apply an ifElse rewrite
else { if .. { .. } } rewrite into else if { .. }
* Use switch-statements over ifElseChains
Rewrite chains of if-else into switch statements. Where applicable,
add an early nil-guard to simplify case analysis. Also, in
ScanTask's Start(..), invert the logic to outdent the whole block, and
help the reader: if it's not a scene, the function flow is now far more
local to the top of the function, and it's clear that the rest of the
function has to do with scene management.
* Enable gocritic on the code base.
Disable appendAssign for now since we aren't passing that check yet.
* Document the nolint additions
* Document StashBoxBatchPerformerTagInput
* Use the request context
The code uses context.Background() in a flow where there is a
http.Request. Use the requests context instead.
* Use a true context in the plugin example
Let AddTag/RemoveTag take a context and use that context throughout
the example.
* Avoid the use of context.Background
Prefer context.TODO over context.Background deep in the call chain.
This marks the site as something which we need to context-handle
later, and also makes it clear to the reader that the context is
sort-of temporary in the code base.
While here, be consistent in handling the `act` variable in each
branch of the if .. { .. } .. check.
* Prefer context.TODO over context.Background
For the different scraping operations here, there is a context
higher up the call chain, which we ought to use. Mark the call-sites
as TODO for now, so we can come back later on a sweep of which parts
can be context-lifted.
* Thread context upwards
Initialization requires context for transactions. Thread the context
upward the call chain.
At the intialization call, add a context.TODO since we can't break this
yet. The singleton assumption prevents us from pulling it up into main for
now.
* make tasks context-aware
Change the task interface to understand contexts.
Pass the context down in some of the branches where it is needed.
* Make QueryStashBoxScene context-aware
This call naturally sits inside the request-context. Use it.
* Introduce a context in the JS plugin code
This allows us to use a context for HTTP calls inside the system.
Mark the context with a TODO at top level for now.
* Nitpick error formatting
Use %v rather than %s for error interfaces.
Do not begin an error strong with a capital letter.
* Avoid the use of http.Get in FFMPEG download chain
Since http.Get has no context, it isn't possible to break out or have
policy induced. The call will block until the GET completes. Rewrite
to use a http Request and provide a context.
Thread the context through the call chain for now. provide
context.TODO() at the top level of the initialization chain.
* Make getRemoteCDPWSAddress aware of contexts
Eliminate a call to http.Get and replace it with a context-aware
variant.
Push the context upwards in the call chain, but plug it before the
scraper interface so we don't have to rewrite said interface yet.
Plugged with context.TODO()
* Scraper: make the getImage function context-aware
Use a context, and pass it upwards. Plug it with context.TODO()
up the chain before the rewrite gets too much out of hand for now.
Minor tweaks along the way, remove a call to context.Background()
deep in the call chain.
* Make NOTIFY request context-aware
The call sits inside a Request-handler. So it's natural to use the
requests context as the context for the outgoing HTTP request.
* Use a context in the url scraper code
We are sitting in code which has a context, so utilize it for the
request as well.
* Use a context when checking versions
When we check the version of stash on Github, use a context. Thread
the context up to the initialization routine of the HTTP/GraphQL
server and plug it with a context.TODO() for now.
This paves the way for providing a context to the HTTP server code in a
future patch.
* Make utils func ReadImage context-aware
In almost all of the cases, there is a context in the call chain which
is a natural use. This is true for all the GraphQL mutations.
The exception is in task_stash_box_tag, so plug that task with
context.TODO() for now.
* Make stash-box get context-aware
Thread a context through the call chain until we hit the Client API.
Plug it with context.TODO() there for now.
* Enable the noctx linter
The code is now free of any uncontexted HTTP request. This means we
pass the noctx linter, and we can enable it in the code base.
* Add collation to directory listings. Closes#1806
Introduce a new `locale` arg to the `Query.directory` field. Set "en"
as the default for the field for backward compatibility. Use the given
locale, sending it through a language matcher, and use `x/text` as the
collation engine for the matched language.
Augment the file `ListDirs` call to optionally take a Collator. If the
Collator is given, sort file listings according to the collators rules.
While here, document the GraphQL schema a bit more.
Add matchers by looking at the current front-end locales, and make sure
each of these occur in the matcher list.
* Language matcher touchups
* Avoid having `en-US` twice.
* Introduce `en-AU`.
* Pass IgnoreCase and Numeric collation
Allow the collator to be configured with options. Pass the options
IgnoreCase and Numeric to the collator.
* Replace error assertions with Go 1.13 style
Use `errors.As(..)` over type assertions. This enables better use of
wrapped errors in the future, and lets us pass some errorlint checks
in the process.
The rewrite is entirely mechanical, and uses a standard idiom for
doing so.
* Use Go 1.13's errors.Is(..)
Rather than directly checking for error equality, use errors.Is(..).
This protects against error wrapping issues in the future.
Even though something like sql.ErrNoRows doesn't need the wrapping, do
so anyway, for the sake of consistency throughout the code base.
The change almost lets us pass the `errorlint` Go checker except for
a missing case in `js.go` which is to be handled separately; it isn't
mechanical, like these changes are.
* Remove goconst
goconst isn't a useful linter in many cases, because it's false positive
rate is high. It's 100% for the current code base.
* Avoid direct comparison of errors in recover()
Assert that we are catching an error from recover(). If we are,
check that the error caught matches errStop.
* Enable the "errorlint" checker
Configure the checker to avoid checking for errorf wraps. These are
often false positives since the suggestion is to blanket wrap errors
with %w, and that exposes the underlying API which you might not want
to do.
The other warnings are good however, and with the current patch stack,
the code base passes all these checks as well.
* Configure rowserrcheck
The project uses sqlx. Configure rowserrcheck to include said package.
* Mechanically rewrite a large set of errors
Mechanically search for errors that look like
fmt.Errorf("...%s", err.Error())
and rewrite those into
fmt.Errorf("...%v", err)
The `fmt` package is error-aware and knows how to call err.Error()
itself.
The rationale is that this is more idiomatic Go; it paves the
way for using error wrapping later with %w in some sites.
This patch only addresses the entirely mechanical rewriting caught by
a project-side search/replace. There are more individual sites not
addressed by this patch.
Reduce allocations. Don't create intermediary arrays which we then
consume right after. Manually fuse the arrays and decode straight into
the sum instead.
Furthermore, don't invoke a Reader, but carve out the locations via a
loop, directly.
These two changes taken together speeds up oshash computations by a
factor of 10 according to the benchmark tests. The main reason for
this change is a much lowered memory allocation rate which in turn
improves GC pressure.
While here, add a benchmark for oshash computations and use it for
testing the performance.
* Refactor scraper structures
* Move matching code into new package
* Add autotag scraper
* Always check first letter of auto-tag names
* Account for nulls
Co-authored-by: Kermie <kermie@isinthe.house>
### **Stash is a self-hosted webapp written in Go which organizes and serves your porn.**
**Stash is a locally hosted web-based app written in Go which organizes and serves your porn.**
* It can gather information about videos in your collection from the internet, and is extensible through the use of community-built plugins for a large number of content producers.
* It supports a wide variety of both video and image formats.
*Stash gathers information about videos in your collection from the internet, and is extensible through the use of community-built plugins for a large number of content producers and sites.
* Stash supports a wide variety of both video and image formats.
* You can tag videos and find them later.
*It provides statistics about performers, tags, studios and other things.
*Stash provides statistics about performers, tags, studios and more.
You can [watch a SFW demo video](https://vimeo.com/545323354) to see it in action.
For further information you can [read the in-app manual](ui/v2.5/src/docs/en).
# Installing stash
# Installing Stash
## via Docker
Follow [this README.md in the docker directory.](docker/production/README.md)
## Pre-Compiled Binaries
The Stash server runs on macOS, Windows, and Linux. Download the [latest release here](https://github.com/stashapp/stash/releases).
Run the executable (double click the exe on windows or run `./stash-osx` / `./stash-linux` from the terminal on macOS / Linux) and navigate to either https://localhost:9999 or http://localhost:9999 to get started.
*Note for Windows users:* Running the app might present a security prompt since the binary isn't yet signed. Bypass this by clicking "more info" and then the "run anyway" button.
Running the app might present a security prompt since the binary isn't yet signed. Bypass this by clicking "more info" and then the "run anyway" button.
#### FFMPEG
If stash is unable to find or download FFMPEG then download it yourself from the link for your platform:
The `ffmpeg(.exe)` and `ffprobe(.exe)` files should be placed in `~/.stash` on macOS / Linux or `C:\Users\YourUsername\.stash` on Windows.
Stash requires ffmpeg. If you don't have it installed, Stash will download a copy for you. It is recommended that Linux users install `ffmpeg` from their distro's package manager.
# Usage
## Quickstart Guide
1) Download and install Stash and its dependencies
2) Run Stash. It will prompt you for some configuration options and a directory to index (you can also do this step afterward)
3) After configuration, launch your web browser and navigate to the URL shown within the Stash app.
Stash is a web-based application. Once the application is running, the interface is available (by default) from http://localhost:9999.
**Note that Stash does not currently retrieve and organize information about your entire library automatically.** You will need to help it along through the use of [scrapers](blob/develop/ui/v2.5/src/docs/en/Scraping.md). The Stash community has developed scrapers for many popular data sources which can be downloaded and installed from [this repository](https://github.com/stashapp/CommunityScrapers).
On first run, Stash will prompt you for some configuration options and media directories to index, called "Scanning" in Stash. After scanning, your media will be available for browsing, curating, editing, and tagging.
The simplest way to tag a large number of files is by using the [Tagger](https://github.com/stashapp/stash/blob/develop/ui/v2.5/src/docs/en/Tagger.md) which uses filename keywords to help identify the file and pull in scene and performer information from our stash-box database. Note that this data source is not comprehensive and you may need to use the scrapers to identify some of your media.
Stash can pull metadata (performers, tags, descriptions, studios, and more) directly from many sites through the use of [scrapers](https://github.com/stashapp/stash/tree/develop/ui/v2.5/src/docs/en/Scraping.md), which integrate directly into Stash.
## CLI
Many community-maintained scrapers are available for download at the [Community Scrapers Collection](https://github.com/stashapp/CommunityScrapers). The community also maintains StashDB, a crowd-sourced repository of scene, studio, and performer information, that can automatically identify much of a typical media collection. Inquire in the Discord for details. Identifying an entire collection will typically require a mix of multiple sources.
Stash runs as a command-line app and local web server. There are some command-line options available, which you can see by running `stash --help`.
<sub>StashDB is the canonical instance of our open source metadata API, [stash-box](https://github.com/stashapp/stash-box).</sub>
For example, to run stash locally on port 80 run it like this (OSX / Linux) `stash --host 127.0.0.1 --port 80`
This command would need customizing for your environment. [This link](https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl) might be useful.
Once you have a certificate and key file name them `stash.crt` and `stash.key` and place them in the same directory as the `config.yml` file, or the `~/.stash` directory. Stash detects these and starts up using HTTPS rather than HTTP.
## Basepath rewriting
The basepath defaults to `/`. When running stash via a reverse proxy in a subpath, the basepath can be changed by having the reverse proxy pass `X-Forwarded-Prefix` (and optionally `X-Forwarded-Port`) headers. When detects these headers, it alters the basepath URL of the UI.
# Customization
## Themes and CSS Customization
There is a [directory of community-created themes](https://github.com/stashapp/stash/wiki/Themes) on our Wiki, along with instructions on how to install them.
You can also make Stash interface fit your desired style with [Custom CSS snippets](https://github.com/stashapp/stash/wiki/Custom-CSS-snippets) and [CSS Tweaks](https://github.com/stashapp/stash/wiki/CSS-Tweaks).
Stash is available in 18 languages (so far!) and it could be in your language too. If you want to help us translate Stash into your language, you can make an account at [translate.stashapp.cc](https://translate.stashapp.cc/projects/stash/stash-desktop-client/) to get started contributing new languages or improving existing ones. Thanks!
# Support (FAQ)
@@ -85,73 +56,18 @@ Answers to other Frequently Asked Questions can be found [on our Wiki](https://g
For issues not addressed there, there are a few options.
* Read the [Wiki](https://github.com/stashapp/stash/wiki)
* Check the in-app documentation (also available [here](https://github.com/stashapp/stash/tree/develop/ui/v2.5/src/docs/en)
* Check the in-app documentation, in the top right corner of the app (also available [here](https://github.com/stashapp/stash/tree/develop/ui/v2.5/src/docs/en)
* Join the [Discord server](https://discord.gg/2TsNFKt), where the community can offer support.
# Compiling From Source Code
# Customization
## Pre-requisites
## Themes and CSS Customization
There is a [directory of community-created themes](https://github.com/stashapp/stash/wiki/Themes) on our Wiki, along with instructions on how to install them.
* [Go](https://golang.org/dl/)
* [GolangCI](https://golangci-lint.run/) - A meta-linter which runs several linters in parallel
* To install, follow the [local installation instructions](https://golangci-lint.run/usage/install/#local-installation)
* Run `yarn install --frozen-lockfile` in the `stash/ui/v2.5` folder (before running make generate for first time).
You can also make Stash interface fit your desired style with [Custom CSS snippets](https://github.com/stashapp/stash/wiki/Custom-CSS-snippets).
NOTE: You may need to run the `go get` commands outside the project directory to avoid modifying the projects module file.
# For Developers
## Environment
Pull requests are welcome!
### macOS
TODO
### Windows
1. Download and install [Go for Windows](https://golang.org/dl/)
2. Download and install [MingW](https://sourceforge.net/projects/mingw-w64/)
3. Search for "advanced system settings" and open the system properties dialog.
1. Click the `Environment Variables` button
2. Under system variables find the `Path`. Edit and add `C:\Program Files\mingw-w64\*\mingw64\bin` (replace * with the correct path).
NOTE: The `make` command in Windows will be `mingw32-make` with MingW.
## Commands
*`make generate` - Generate Go and UI GraphQL files
*`make build` - Builds the binary (make sure to build the UI as well... see below)
*`make docker-build` - Locally builds and tags a complete 'stash/build' docker image
*`make pre-ui` - Installs the UI dependencies. Only needs to be run once before building the UI for the first time, or if the dependencies are updated
*`make fmt-ui` - Formats the UI source code
*`make ui` - Builds the frontend
*`make lint` - Run the linter on the backend
*`make fmt` - Run `go fmt`
*`make it` - Run the unit and integration tests
*`make validate` - Run all of the tests and checks required to submit a PR
*`make ui-start` - Runs the UI in development mode. Requires a running stash server to connect to. Stash port can be changed from the default of `9999` with environment variable `REACT_APP_PLATFORM_PORT`.
## Building a release
1. Run `make generate` to create generated files
2. Run `make ui` to compile the frontend
3. Run `make build` to build the executable for your current platform
## Cross compiling
This project uses a modification of the [CI-GoReleaser](https://github.com/bep/dockerfiles/tree/master/ci-goreleaser) docker container to create an environment
where the app can be cross-compiled. This process is kicked off by CI via the `scripts/cross-compile.sh` script. Run the following
command to open a bash shell to the container to poke around:
Stash can be profiled using the `--cpuprofile <output profile filename>` command line flag.
The resulting file can then be used with pprof as follows:
`go tool pprof <path to binary> <path to profile filename>`
With `graphviz` installed and in the path, a call graph can be generated with:
`go tool pprof -svg <path to binary> <path to profile filename> > <output svg file>`
See [Development](docs/DEVELOPMENT.md) and [Contributing](docs/CONTRIBUTING.md) for information on working with the codebase, getting a local development setup, and contributing changes.
This dockerfile is used to build a stash docker container using the current source code.
This dockerfile is used to build a stash docker container using the current source code. This is ideal for testing your current branch in docker. Note that it does not include python, so python-based scrapers will not work in this image. The production docker images distributed by the project contain python and the necessary packages.
# Building the docker container
From the top-level directory (should contain `main.go` file):
Modified from https://github.com/bep/dockerfiles/tree/master/ci-goreleaser
When the dockerfile is changed, the version number should be incremented in the Makefile and the new version tag should be pushed to docker hub. The `scripts/cross-compile.sh` script should also be updated to use the new version number tag, and `.travis.yml` needs to be updated to pull the correct image tag.
A MacOS univeral binary can be created using `lipo -create -output stash-osx-universal stash-osx stash-osx-applesilicon`, available in the image.
When the dockerfile is changed, the version number should be incremented in the Makefile and the new version tag should be pushed to docker hub. The `scripts/cross-compile.sh` script should also be updated to use the new version number tag, and the github workflow files need to be updated to pull the correct image tag.
@@ -7,16 +7,6 @@ Only `docker` and `docker-compose` are required. For the most part your understa
Installation instructions are available below, and if your distrobution's repository ships a current version of docker, you may use that.
https://docs.docker.com/engine/install/
### Docker
Docker is effectively a cross-platform software package repository. It allows you to ship an entire environment in what's referred to as a container. Containers are intended to hold everything that is needed to run an application from one place to another, making it easy for everyone along the way to reproduce the environment.
The StashApp docker container ships with everything you need to automatically build and run stash, including ffmpeg.
### docker-compose
Docker Compose lets you specify how and where to run your containers, and to manage their environment. The docker-compose.yml file in this folder gets you a fully working instance of StashApp exactly as you would need it to have a reasonable instance for testing / developing on. If you are deploying a live instance for production, a reverse proxy (such as NGINX or Traefik) is recommended, but not required.
The latest version is always recommended.
### Get the docker-compose.yml file
Now you can either navigate to the [docker-compose.yml](https://raw.githubusercontent.com/stashapp/stash/master/docker/production/docker-compose.yml) in the repository, or if you have curl, you can make your Linux console do it for you:
@@ -35,3 +25,13 @@ docker-compose up -d
Installing StashApp this way will by default bind stash to port 9999. This is available in your web browser locally at http://localhost:9999 or on your network as http://YOUR-LOCAL-IP:9999
Good luck and have fun!
### Docker
Docker is effectively a cross-platform software package repository. It allows you to ship an entire environment in what's referred to as a container. Containers are intended to hold everything that is needed to run an application from one place to another, making it easy for everyone along the way to reproduce the environment.
The StashApp docker container ships with everything you need to automatically build and run stash, including ffmpeg.
### docker-compose
Docker Compose lets you specify how and where to run your containers, and to manage their environment. The docker-compose.yml file in this folder gets you a fully working instance of StashApp exactly as you would need it to have a reasonable instance for testing / developing on. If you are deploying a live instance for production, a reverse proxy (such as NGINX or Traefik) is recommended, but not required.
* Run `yarn install --frozen-lockfile` in the `stash/ui/v2.5` folder (before running make generate for first time).
NOTE: You may need to run the `go get` commands outside the project directory to avoid modifying the projects module file.
## Environment
### Windows
1. Download and install [Go for Windows](https://golang.org/dl/)
2. Download and extract [MingW64](https://sourceforge.net/projects/mingw-w64/files/) (scroll down and select x86_64-posix-seh, dont use the autoinstaller it doesnt work)
3. Search for "advanced system settings" and open the system properties dialog.
1. Click the `Environment Variables` button
2. Under system variables find the `Path`. Edit and add `C:\MinGW\bin` (replace with the correct path to where you extracted MingW64).
NOTE: The `make` command in Windows will be `mingw32-make` with MingW. For example `make pre-ui` will be `mingw32-make pre-ui`
### macOS
1. If you don't have it already, install the [Homebrew package manager](https://brew.sh).
2. Install dependencies: `brew install go git yarn gcc make`
## Commands
*`make pre-ui` - Installs the UI dependencies. Only needs to be run once before building the UI for the first time, or if the dependencies are updated
*`make generate` - Generate Go and UI GraphQL files
*`make fmt-ui` - Formats the UI source code
*`make ui` - Builds the frontend
*`make build` - Builds the binary (make sure to build the UI as well... see below)
*`make docker-build` - Locally builds and tags a complete 'stash/build' docker image
*`make lint` - Run the linter on the backend
*`make fmt` - Run `go fmt`
*`make it` - Run the unit and integration tests
*`make validate` - Run all of the tests and checks required to submit a PR
*`make ui-start` - Runs the UI in development mode. Requires a running stash server to connect to. Stash server port can be changed from the default of `9999` using environment variable `VITE_APP_PLATFORM_PORT`. UI runs on port `3000` or the next available port.
## Building a release
1. Run `make pre-ui` to install UI dependencies
2. Run `make generate` to create generated files
3. Run `make ui` to compile the frontend
4. Run `make build` to build the executable for your current platform
## Cross compiling
This project uses a modification of the [CI-GoReleaser](https://github.com/bep/dockerfiles/tree/master/ci-goreleaser) docker container to create an environment
where the app can be cross-compiled. This process is kicked off by CI via the `scripts/cross-compile.sh` script. Run the following
command to open a bash shell to the container to poke around:
scrapeFreeones(performer_name:String!):ScrapedPerformer@deprecated(reason:"use scrapeSinglePerformer with scraper_id = builtin_freeones")
"""Scrape a list of performers from a query"""
scrapeFreeonesPerformerList(query:String!):[String!]!@deprecated(reason:"use scrapeSinglePerformer with scraper_id = builtin_freeones")
"""Query StashBox for scenes"""
queryStashBoxScene(input:StashBoxSceneQueryInput!):[ScrapedScene!]!@deprecated(reason:"use scrapeSingleScene or scrapeMultiScenes")
"""Query StashBox for performers"""
queryStashBoxPerformer(input:StashBoxPerformerQueryInput!):[StashBoxPerformerQueryResult!]!@deprecated(reason:"use scrapeSinglePerformer or scrapeMultiPerformers")
# === end deprecated methods ===
# Plugins
"""List loaded plugins"""
plugins:[Plugin!]
@@ -127,7 +125,13 @@ type Query {
"""Returns the current, complete configuration"""
configuration:ConfigResult!
"""Returns an array of paths for the given path"""
directory(path:String):Directory!
directory(
"The directory path to list"
path:String,
"Desired collation locale. Determines the order of the directory result. eg. 'en-US', 'pt-BR', ..."
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
