diff --git a/doc/dev/convert-csv-to-yml.py b/doc/convert-csv-to-yml.py similarity index 60% rename from doc/dev/convert-csv-to-yml.py rename to doc/convert-csv-to-yml.py index e3d1347..0f283b5 100644 --- a/doc/dev/convert-csv-to-yml.py +++ b/doc/convert-csv-to-yml.py @@ -1,3 +1,5 @@ +# this temporary script converts .csv to .yml +# can delete when documentation migration is done import sys,csv r = csv.reader(open(sys.argv[1])) keys = next(r) diff --git a/doc/dev/pitfalls.md b/doc/dev/coding-tips.md similarity index 51% rename from doc/dev/pitfalls.md rename to doc/dev/coding-tips.md index d179c7f..07bb0b5 100644 --- a/doc/dev/pitfalls.md +++ b/doc/dev/coding-tips.md @@ -1,20 +1,8 @@ -## Strict CSP +Some not-so-obvious tricks. -Reference: search for "Content-Security-Policy" in **.go +## Treat cookies as user-provided values -Current CSP disallows inline styles and scripts and iframes. - -## Low Quality Go Module: net/url - -`url.Path` is stored decoded (no %XX). `url.Scheme` is stored without `://` (mandated by RFC). Not sure why Go does that. Felt like this is bound to cause some nasty bug on decoding and encoding. - -Current proxied URLs don't have weird characters in them. Hopefully it stays this way. - -Solution: Replace "net/url" with a better third-party module - -## Jet templating engine is not type checked - -Solution: [templ](https://github.com/a-h/templ) +I think we do pretty well here. ## Jet `.Illust` vs `Illust` diff --git a/doc/dev/design-flaws.md b/doc/dev/design-flaws.md new file mode 100644 index 0000000..c84a9ee --- /dev/null +++ b/doc/dev/design-flaws.md @@ -0,0 +1,15 @@ +# Current Design Flaws + +This section documents some bad/buggy designs in PixivFE's design, both frontend and backend. + +## "Switchers" + +Switchers should know what the current value is. + +## Low Quality Go Module: net/url + +`url.Path` is stored decoded (no %XX). `url.Scheme` is stored without `://` (mandated by RFC). Not sure why Go does that. Felt like this is bound to cause some nasty bug on decoding and encoding. + +Current proxied URLs don't have weird characters in them. Hopefully it stays this way. + +Solution: Replace "net/url" with a better third-party module diff --git a/doc/dev/unified.md b/doc/dev/feature-ideas.md similarity index 79% rename from doc/dev/unified.md rename to doc/dev/feature-ideas.md index 3a8a08f..ddc4cdc 100644 --- a/doc/dev/unified.md +++ b/doc/dev/feature-ideas.md @@ -1,10 +1,8 @@ -# A unified documentation for PixivFE - -## Proposals +# Proposals This section contains some potential features or redesigns that can be implemented into PixivFE. -### Pixivision +## Pixivision **Summary**: Pixivision is a service owned by Pixiv that publishes articles about various types of artwork themes... **Notes**: [Pixivision](https://www.pixivision.net/en/) is an independent service. Most of the data comes from Pixivision's website. @@ -14,7 +12,7 @@ Pixivision does not provide an API for data access. Pages on Pixivision seems to - Write a separate module for Pixivision and integrate it into PixivFE. - We do web scraping and HTML parsing for this one. -### Sketch +## Sketch **Summary**: Sketch is a service owned by Pixiv that allow users to livestream, mostly for their drawing process. @@ -26,7 +24,7 @@ Detailed notes TBA. - Thanks to the public API, pages could be build easily. - For the streaming part, we may have to include a JavaScript library for HLS streaming. -### Ugoira support +## Ugoira support **Summary**: Ugoiras are Pixiv's "animated image" format. @@ -39,7 +37,7 @@ You can checkout Pixiv's implementation on their own ugoira player [here](https: - GIF/APNG/WEBP renderer. - Some people want to convert ugoiras to video formats? (no idea) -### Landing page +## Landing page **Summary**: PixivFE's homepage. @@ -51,7 +49,7 @@ The only thing left is to write the frontend for them. Detailed notes TBA. - Spend some time to write some HTML/SCSS. - Currently, you have to authenticate (login) in order to access the *full* landing page. Can we show the *full* page to unauthenticated users as well? -### Popular artworks +## Popular artworks **Summary**: Pixiv has a "Sort by views" and "Sort by bookmarks" feature that is only available for premium users. @@ -61,7 +59,7 @@ The only thing left is to write the frontend for them. Detailed notes TBA. - Look into repos that attempts to retrieve popular artworks - If search "hacking" is possible, could there be more "hacks" around? -### "User discovery" page +## "User discovery" page **Summary**: Like artwork discovery, but it is for users. @@ -71,7 +69,7 @@ The development for this page has been put on hold because of it, since "followi **Ideas**: - It is easy to implement thanks to the API. -### Search suggestions +## Search suggestions **Summary**: Pixiv provides [an API endpoint](https://www.pixiv.net/ajax/search/suggestion?mode=all&lang=en) for search suggestions. @@ -82,7 +80,7 @@ The development for this page has been put on hold because of it, since "followi - We can implement JavaScript to fetch the suggestions every time the user focuses on the search bar. - We can create a separate page just for this. -### App API support +## App API support **Summary**: Apart of the public AJAX API, Pixiv also provides a private API, used specifically for mobile applications. @@ -92,37 +90,15 @@ I added this section because there are some limitations to the public API (follo **Ideas**: - Write more stuff when desperate. -### Novel page -### Image grid layout -### Series -### Server's PixivFE Git version/commit -### Artwork filters -### Quick download button -### Download all buttons for all containers -### Filters -### Page profile -### Option to open in new tabs -### Download button in artwork page -### Option to select default image quality in artwork page - -## Flaws - -This section documents some bad/buggy designs in PixivFE's design, both frontend and backend. - -### Cookies management / validation - -All cookie values could be set directly without validation. Should we be concerned? - -### "Switchers" - -Switchers should know what the current value is. - -### "kmutex" - -Sometimes panics. - -## References - -This section contains multiple external links to materials/resources that could help. - -- [pixiv.pics](https://www.pixiv.pics/) +## Novel page +## Image grid layout +## Series +## Server's PixivFE Git version/commit +## Artwork filters +## Quick download button +## Download all buttons for all containers +## Filters +## Page profile +## Option to open in new tabs +## Download button in artwork page +## Option to select default image quality in artwork page diff --git a/doc/dev/features/caching.md b/doc/dev/features/caching.md new file mode 100644 index 0000000..7735279 --- /dev/null +++ b/doc/dev/features/caching.md @@ -0,0 +1,40 @@ +I thought about caching a little bit. + +## Guide for caching + +The majority of the site's traffic is images, and all images on the Pixiv route can be cached forever. + Cached in browser: pixiv headers cache images for year + On the server, the reverse proxy (e.g. nginx) can be cached worry-free forever. + +JSON requests to Pixiv can be cached for some time. + todo: We should have a middleware handling that. + +Assets are already cached properly thanks to net/http. + +Every rendered page can be cached for 5 minutes or so in the browser. +However, after changing settings, the page need to be refreshed instantly. + Cache in browser: todo: read below implement in template/render.go + The reverse proxy shouldn't do anything about this. + +The only three headers that controls caching in browser and middleboxes: + +- [Age](https://httpwg.org/specs/rfc9111.html#field.age) +- [Cache-Control](https://httpwg.org/specs/rfc9111.html#field.cache-control) +- [Expires](https://httpwg.org/specs/rfc9111.html#field.expires) + +## Predictive Caching + +Preload header is already implemented for /artwork/ and /artwork-multi/ for the main images. + +However, we can do better. + +When we send an HTML page to the client, we can start fetching all the images in the page. We need a lot of correct synchronization to get it right, but it will make the browsing experience faster. + +- pixivfe predictively fetch images to store in its cache. +- every proxied request goes through the cache as well. + +every cache item has three possible states: + +- not in cache +- pending +- cached diff --git a/doc/dev/novels.md b/doc/dev/features/novels.md similarity index 100% rename from doc/dev/novels.md rename to doc/dev/features/novels.md diff --git a/doc/dev/user-customization.md b/doc/dev/features/user-customization.md similarity index 100% rename from doc/dev/user-customization.md rename to doc/dev/features/user-customization.md diff --git a/doc/dev/framework-migration.md b/doc/dev/framework-migration.md index 5d19436..33384e9 100644 --- a/doc/dev/framework-migration.md +++ b/doc/dev/framework-migration.md @@ -12,27 +12,11 @@ ## todo -- add limiter (maybe it should be in nginx) -- add caching (maybe it should be in nginx) - -## Guide for caching - -The majority of the site's traffic is images, and all images on the Pixiv route can be cached forever. - Cached in browser: pixiv headers cache images for year - On the server, the reverse proxy (e.g. nginx) can be cached worry-free forever. - -JSON requests to Pixiv can be cached for some time. - todo: We should have a middleware handling that. - -Assets are already cached properly thanks to net/http. - -Every rendered page can be cached for 5 minutes or so in the browser. -However, after changing settings, the page need to be refreshed instantly. - Cache in browser: todo: read below implement in template/render.go - The reverse proxy shouldn't do anything about this. - -The only three headers that controls caching in browser and middleboxes: - -- [Age](https://httpwg.org/specs/rfc9111.html#field.age) -- [Cache-Control](https://httpwg.org/specs/rfc9111.html#field.cache-control) -- [Expires](https://httpwg.org/specs/rfc9111.html#field.expires) +- [x] [add caching](features/caching.md) +- [ ] add limiter (maybe it should be in nginx) +- check if everything works + - [x] templating (this has tests) + - [x] search, artworks, users + - [x] novels and novel settings + - [x] settings + - what else? diff --git a/doc/dev/helpful-resources.md b/doc/dev/helpful-resources.md new file mode 100644 index 0000000..f3fb80a --- /dev/null +++ b/doc/dev/helpful-resources.md @@ -0,0 +1,6 @@ +# Reference Material + +This section contains multiple external links to materials/resources that could help. + +- [Pixiv Viewer - Yet Another Pixiv Illustration & Novel Viewer](https://www.pixiv.pics/) + diff --git a/doc/dev/general.md b/doc/dev/roadmap.md similarity index 94% rename from doc/dev/general.md rename to doc/dev/roadmap.md index 03de36e..ef08c1e 100644 --- a/doc/dev/general.md +++ b/doc/dev/roadmap.md @@ -1,16 +1,18 @@ # Roadmap +Planned features. + ## To implement /settings/ - [x] Merge login page with settings page - [x] Persistence (http-only secure cookies) -- [User Settings](user-customization.md) +- [User Settings](features/user-customization.md) /novel/ -- [Novel support](novels.md) +- [Novel support](features/novels.md) Might need some ideas for the reader's UI. Allow options for font size and family? Black and white backgrounds? diff --git a/routes/artworkMulti.go b/routes/artworkMulti.go index a657c83..0fae161 100644 --- a/routes/artworkMulti.go +++ b/routes/artworkMulti.go @@ -58,6 +58,12 @@ func ArtworkMultiPage(w http.ResponseWriter, r *http.Request) error { return err_global } + for _, illust := range artworks { + for _, img := range illust.Images { + PreloadImage(w, img.Large) + } + } + return Render(w, r, Data_artworkMulti{ Artworks: artworks, Title: fmt.Sprintf("(%d images)", len(artworks)),