Cutscene upscaling

Improving the artwork in Warzone2100 - not for mod discussions
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

Upscaled backgrounds: https://drive.google.com/open?id=1zzAK1 ... a2SySqcuZu Feel free to edit/use them as you wish.

Took a pause on trying to train a custom model. The weather is too warm for running my GPU on full for days in a row..
aliswe
Trained
Trained
Posts: 63
Joined: 01 Apr 2020, 22:11

Re: Cutscene upscaling

Post by aliswe »

Great, I'll submit them in the PR.

THANKS!
aliswe
Trained
Trained
Posts: 63
Joined: 01 Apr 2020, 22:11

Re: Cutscene upscaling

Post by aliswe »

These really are amazing. How did you do this?! Added for including.

Only one is left though (missionend.png), something you want to do? or rather not?
User avatar
AlexTheDacian
Rookie
Rookie
Posts: 16
Joined: 17 Feb 2020, 12:33

Re: Cutscene upscaling

Post by AlexTheDacian »

Okay Phase II, we got it, you're amazing. Just kidding. :P

The cutscenes and the backgrounds are incredible. Keep up the good work!
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

That image had much lower quality. I think I have repaired as much as I can on it: https://drive.google.com/open?id=1VtQ4e ... lpvEZKLt_w

For individual images, I use esrgan with multiple different models (upscale model, then downsample; downsample, then upscale model; artifact removal model, then upscale model, then downsample; ...). Then combine all the resulting images through some combination of pixel averaging, median blending, and manually selecting patches. This aims to get the best of what each separate model can produce.

For cutscenes, I automate more (it is not reasonable to manually edit every frame...). Split the video into frames. Divide the frames into categories (e.g. logo, map, animation...). Process each category with the model series that seems to give the best results. Render the frames back to a video.

Later this year, I aim to improve my cutscene upscales. My trained models look like they will be yielding better results, but there are still some quirks to work out of them. It just is too warm to keep training them right now.
pastdue
Warzone 2100 Team Member
Warzone 2100 Team Member
Posts: 339
Joined: 13 Aug 2017, 17:44

Re: Cutscene upscaling

Post by pastdue »

Phase II wrote: 09 May 2020, 04:40Later this year, I aim to improve my cutscene upscales. My trained models look like they will be yielding better results, but there are still some quirks to work out of them. It just is too warm to keep training them right now.
Thank you for all of your efforts. These really are exceptional improvements / upscaling.

As a heads-up: I hope we can transition from the current video format to VP9 + WebM container in the future.

Not sure when precisely that will happen, but just wanted to make you aware since you are in the process of your upscaling and conversion. Perhaps we can upload the raw(?) format somewhere to reconvert / recompress in the future. (What initial output formats does your pipeline generate?)
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

Intermediate data is one png per frame. Frame sizes range from around 100 KB to 1.5 MB. Size adds up quickly after a couple clips. I could pack them in a lossless VP9 (uncertain how much that will shave off the size), to support future re-encoding usage.
User avatar
Ezio
Trained
Trained
Posts: 306
Joined: 24 Apr 2010, 16:42

Re: Cutscene upscaling

Post by Ezio »

soo I tried doing the same thing, made several versions of a single video with a different ai model, AA, denoise, deblock, and stuff. but it still gets me that I'd prefer the remakes instead.

the link: https://mega.nz/folder/pdFWzZLR#rjr0TtRiaeIDNTporWBCVA
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

So, I have not forgotten about this project. Training a model is easy (ish... takes 1-4 days to for the neural network to converge, depending on difficulty of what it is trying to predict). Training a good model is hard (highly sensitive to input data). Training a model to handle an unknown compression codec, with maybe 30 high quality reference images? Very hard. That said, I think I am nearing production ready.

I found an original iso on the internet archive. Videos were *.rpl, which ffmpeg fortunately can read (unfortunately cannot write, otherwise I would be able to generate a large data set). They contain greater detail than the .ogg files currently shipped. However, they also contain irregular burst artifacts. These have been a struggle to handle.

Input rpl frame:
Image

Direct upscale:
Image

Prefilter, then upscale:
Image
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

The prefilter is a model trained by me hand editing frames (takes me about 10 minutes per frame). As you can tell, it handles some, but not all of the distortions. Opinions on if this is sufficient? Should I try improving the prefilter?

Alternatively, I could work from the ogg files. They do not have this issue, but give overall worse results due to more degradation than the rpl files.
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

Managed to find some .avi files. They appear to have similar detail level as the .rpl files, but without the irregular artifacts. Looks like it is back to training models for me...

pastdue wrote: 09 May 2020, 23:09 [...] Perhaps we can upload the raw(?) format somewhere to reconvert / recompress in the future. (What initial output formats does your pipeline generate?)
Did some tests. I estimate lossless compressed version will weigh in around 17 GB. Any ideas on the best way will be for me to distribute those? I suppose could upload a batch, wait for people to download, delete it and upload the next batch.
User avatar
AlexTheDacian
Rookie
Rookie
Posts: 16
Joined: 17 Feb 2020, 12:33

Re: Cutscene upscaling

Post by AlexTheDacian »

Best way would be on a MEGA free account, as the speeds are relatively high, and the space that you're allowed to have is quite decent: between 15 and 50 GBs
I'mma big Lupus now :lol2:
pastdue
Warzone 2100 Team Member
Warzone 2100 Team Member
Posts: 339
Joined: 13 Aug 2017, 17:44

Re: Cutscene upscaling

Post by pastdue »

This is fantastic.

Also, Phase II, if you haven't checked out the Discord yet - we have some people there who are also eager to help out with upscaling / fixing artifacts / etc, so if you have time to pop in we could get a good group-chat going:

https://discord.com/invite/ZvRVQ8g

EDIT: And yes, the MEGA route could work (or whatever is most convenient), and then we can mirror them elsewhere. If individual video files are below 5GB, you could upload them to - for example - a new SourceForge project.
Phase II
Trained
Trained
Posts: 32
Joined: 14 Aug 2016, 01:22

Re: Cutscene upscaling

Post by Phase II »

Completed ESRGAN model for processing the AVI versions of the cutscenes: https://drive.google.com/file/d/1--Xb20 ... sp=sharing

I found the AVIs from the link mentioned in viewtopic.php?f=30&t=5127&start=15#p118724 (had to use the internet archive).

Now to do the actual upscaling...
Post Reply