JavisDiT 0904 — fps-fixed evaluation results

All numbers use the fps-fixed test split (inpainted side re-timed to the source frame rate). Best value per column is highlighted. Video: 256×256 / 16 fps / 81 frames. Audio: 16 kHz, 5.0625 s. DeSync: Synchformer offset (s), lower = better sync.

1 · Full fps-fixed test set (1,425 clips)

Object removal — video (target: fps-fixed object-removed clip)

method	n	PSNR↑	SSIM↑	LPIPS↓	FVD↓
do nothing (input)	1425	18.67	0.699	0.279	521.6
ROSE	1425	20.30	0.719	0.239	176.0
VideoPainter	1425	18.43	0.676	0.303	362.3
LGVI	1425	16.22	0.546	0.402	402.7
AVI-Edit	1368	16.65	0.640	0.323	580.5
InstructAV2AV	1419	18.88	0.683	0.288	180.1
OURS mainline ep018	1425	20.64	0.677	0.267	128.7
noref ep010	1425	20.09	0.665	0.278	206.0
remove-only ep021	1425	20.20	0.666	0.278	182.1
srcattn ep006	1425	13.85	0.397	0.569	459.9

Mainline ep018 vs remove-only ep021 on removal, paired over 1,425 clips, with 50 examples: open →

Object addition — video (target: original clip with the object)

method	n	PSNR↑	SSIM↑	LPIPS↓	FVD↓
do nothing (input)	1425	18.67	0.699	0.279	521.6
AVI-Edit	1368	16.60	0.636	0.281	387.2
AVI-Edit (1st frame)	1425	17.92	0.661	0.243	293.5
InstructAV2AV (1st frame)	1425	17.01	0.611	0.292	202.5
OURS mainline ep010 (1st frame)	1425	17.24	0.540	0.303	976.5
noref ep010	1425	16.65	0.568	0.319	537.7
srcattn ep006 (1st frame)	1425	12.03	0.301	0.567	933.6

A/V synchronisation — DeSync ↓ (Synchformer, 5 s)

GT (VAE round-trip) = the GT latents decoded by the same VAE as our outputs. In removal, DeSync barely separates methods (the GT itself scores ~0.82).

removal

method	DeSync↓	n
VideoPainter + ZEUS	0.698	1425
ROSE + ZEUS	0.757	1425
LGVI + ZEUS	0.934	1425
AVI-Edit (oracle audio)	0.872	1368
GT video + CoherentAVEdit	0.831	1425
InstructAV2AV	0.893	1419
OURS mainline ep018	0.844	1425
noref ep010	0.846	1425
remove-only ep021	0.850	1425
srcattn ep006	0.890	1425
GT (raw)	0.824	1425
GT (VAE round-trip)	0.838	1425

addition

method	DeSync↓	n
AVI-Edit (oracle audio)	0.767	1368
AVI-Edit (1st frame, oracle audio)	0.753	1425
InstructAV2AV (1st frame)	0.787	1425
GT video + CoherentAVEdit	0.756	1425
OURS mainline ep010 (1st frame)	0.999	1425
noref ep010	0.993	1425
srcattn ep006 (1st frame)	1.037	1425
GT (raw)	0.537	1425
GT (VAE round-trip)	0.599	1425

Addition diagnosis (swap test, 1,425): GT video+GT audio 0.60 · input 0.84 · GT video+OURS audio 0.88 · OURS video+GT audio 0.96 · OURS video+OURS audio 1.00 → the generated add video is what breaks sync (added object motion does not match its sound).

Audio (1,425; FAD on the login CPU) — baselines only

Our models' full-test audio metrics have not been computed yet; see the 50-clip tables below.

method	n	FAD↓	KL(sig)↓	IS↑	LSD↓
do nothing (removal input)	1425	6.98	0.129	1.01	12.95
ZEUS (AudioEditingCode)	1425	1.66	0.204	1.01	14.11
AUDIT	1425	11.23	0.249	1.01	16.60
InstructAV2AV (removal)	1419	0.89	0.143	1.01	14.71
CoherentAVEdit (removal)	1425	1.46	0.218	1.01	15.07
CoherentAVEdit (addition, vs full mix)	1425	5.32	0.189	1.01	15.71

2 · refwarm100k ep014 on 50 test clips (test_fpsfix_rand50_seed0)

refwarm100k keeps the first-frame anchor in training until step 100k (p_ref ≈ 0.55 at ep014), so the anchor is in-distribution; mainline annealed it to 0 by step 30k. FVD/FAD on 50 clips are strongly biased upward (the do-nothing row is 1543 here vs 522 on 1,425) — read the per-clip columns.

Addition (with first frame)

method	PSNR↑	SSIM↑	LPIPS↓	FVD↓	FAD↓	KL(sig)↓	IS↑	LSD↓	DeSync↓
refwarm100k ep014 (1st frame)	18.71	0.605	0.248	1249.4	10.28	0.101	1.01	11.50	1.072
mainline ep010 (1st frame)	16.88	0.522	0.323	2290.1	10.32	0.103	1.01	11.77	0.912
noref ep010	16.26	0.534	0.348	1755.5	10.85	0.112	1.01	11.40	1.064
InstructAV2AV (1st frame)	16.15	0.565	0.327	1258.2	12.62	0.241	1.01	17.37	0.680
AVI-Edit (1st frame, oracle audio)	17.38	0.600	0.282	1399.2	–	–	–	–	–
do nothing (input)	18.05	0.642	0.335	1542.7	–	–	–	–	–
GT (VAE round-trip)	–	–	–	–	3.11	0.027	1.01	8.58	0.684
GT (raw)	–	–	–	–	–	–	–	–	0.664

Paired DeSync (Wilcoxon), refwarm ep014 vs

vs	n	mean Δ (refwarm − other)	refwarm better / worse	p
mainline ep010 (1st frame)	50	+0.160	14 / 21	0.209
noref ep010	50	+0.008	23 / 19	0.788
InstructAV2AV (1st frame)	50	+0.392	15 / 30	0.00987
GT (VAE round-trip)	50	+0.388	12 / 32	0.00357
GT (raw)	50	+0.408	11 / 34	0.0041

Removal (no first frame)

method	PSNR↑	SSIM↑	LPIPS↓	FVD↓	FAD↓	KL(sig)↓	IS↑	LSD↓	DeSync↓
refwarm100k ep014 (no anchor)	18.11	0.617	0.346	1151.7	6.52	0.151	1.01	13.96	0.656
mainline ep018 (no anchor)	19.74	0.647	0.305	918.9	6.78	0.150	1.01	13.57	0.960
noref ep010	19.03	0.634	0.319	1106.9	6.15	0.164	1.01	13.77	0.780
remove-only ep021	18.85	0.636	0.312	1036.5	6.29	0.174	1.01	13.71	0.688
InstructAV2AV	18.41	0.625	0.328	1124.2	5.15	0.123	1.01	14.50	0.861
ROSE (video only)	19.01	0.650	0.301	1099.6	–	–	–	–	–
CoherentAVEdit (audio only)	–	–	–	–	4.50	0.199	1.01	15.45	–
do nothing (input)	18.05	0.642	0.335	1542.7	7.84	0.109	1.01	13.15	–
GT (VAE round-trip)	–	–	–	–	2.46	0.029	1.01	7.88	0.764
GT (raw)	–	–	–	–	–	–	–	–	0.768

Paired DeSync (Wilcoxon), refwarm ep014 vs

vs	n	mean Δ (refwarm − other)	refwarm better / worse	p
mainline ep018 (no anchor)	50	-0.304	24 / 14	0.00589
noref ep010	50	-0.124	20 / 12	0.095
remove-only ep021	50	-0.032	15 / 14	0.705
InstructAV2AV	49	-0.204	21 / 14	0.0852
GT (VAE round-trip)	50	-0.108	18 / 18	0.226
GT (raw)	50	-0.112	17 / 19	0.469

Removal: refwarm has the best DeSync but the lowest PSNR/SSIM — its output drifts further from the input (L1 to input 26.4 vs mainline 22.4), i.e. lower fidelity rather than incomplete removal.

2b · Gemini A/V-consistency judge (499 test clips, gemini-3.1-pro-preview)

Yes-rate in %; all3 = all three questions Yes. p = exact McNemar on all3 vs mainline (paired). Mainline = ep018 removal / ep010 addition with first frame; noref ep010 has no anchor in either direction. InstructAV2AV removal answers are reused from the 09-16 run (byte-identical inputs); its addition is re-judged on the fps-fixed first-frame videos.

Caveat: these are audio-visual agreement scores, not edit-success scores. The judge prompt tells Gemini not to judge whether the instruction was followed. q1 is Yes when the object is both seen and heard, or neither seen nor heard, so an unedited clip can score Yes. A q1 No in removal typically means the object left the picture but its sound remained (or vice versa).

remove

row	n	q1 object	q2 other sources	q3 timing	all3	p all3 vs mainline
mainline	499	53.1	80.2	82.0	45.3	–
noref_ep010	499	56.7	82.6	81.8	50.9	0.03
removeonly_ep021	499	53.5	78.8	78.8	46.1	0.80
InstructAV2AV	497	61.6	82.5	82.3	56.5	8.41e-05

add

row	n	q1 object	q2 other sources	q3 timing	all3	p all3 vs mainline
mainline	499	72.7	80.0	78.0	66.3	–
noref_ep010	499	60.3	72.3	71.3	51.7	9.74e-08
InstructAV2AV	499	60.3	71.9	68.9	51.9	5.86e-07

3 · Checkpoint checks on validation clips (seed-paired)

New: refwarm100k ep018 vs mainline ep018 on 100 test clips (metrics + all 100 examples, with InstructAV2AV): open →

Addition successes: 50 random clips where refwarm ep019 passes all three Gemini questions (406 of 1,425): open →

refwarm100k: ep014 vs ep015 on 100 val clips (val_fpsfix_rand100_seed0) — keep training? 30 examples →

cell · ckpt	PSNR↑	SSIM↑	LPIPS↓	FVD↓	FAD↓	KL↓	LSD↓
noref_remove · ep014	19.27	0.655	0.313	616	0.53	1.300	14.08
noref_remove · ep015	19.15	0.627	0.317	883	1.03	1.401	14.76
ref_add · ep014	18.50	0.613	0.249	924	0.31	1.129	12.34
ref_add · ep015	18.36	0.590	0.260	1126	0.37	1.140	12.45

Paired per-clip change, later − earlier (later better / worse, Wilcoxon p).

cell	ΔPSNR	ΔSSIM	ΔLPIPS	ΔKL	ΔLSD
noref_remove: ep015 − ep014	-0.121 (39/61, p=0.080)	-0.028 (20/80, p=0.000)	+0.004 (41/59, p=0.051)	+0.101 (47/53, p=0.741)	+0.677 (51/49, p=0.575)
ref_add: ep015 − ep014	-0.139 (34/66, p=0.015)	-0.023 (19/81, p=0.000)	+0.011 (26/74, p=0.000)	+0.011 (53/47, p=0.693)	+0.105 (57/43, p=0.417)

mainline: ep018 vs ep021 (50 val clips)

cell · ckpt	PSNR↑	SSIM↑	LPIPS↓	FVD↓	FAD↓	KL↓	LSD↓
noref_remove · ep018	21.06	0.693	0.272	682	1.09	1.323	13.64
noref_remove · ep021	20.77	0.685	0.271	713	1.30	1.340	13.82
ref_add · ep018	16.56	0.565	0.318	1641	0.67	1.141	12.00
ref_add · ep021	16.47	0.553	0.322	1647	0.68	0.965	11.90

Paired per-clip change, later − earlier (later better / worse, Wilcoxon p).

cell	ΔPSNR	ΔSSIM	ΔLPIPS	ΔKL	ΔLSD
noref_remove: ep021 − ep018	-0.292 (23/27, p=0.674)	-0.008 (16/34, p=0.036)	-0.000 (25/25, p=0.653)	+0.017 (27/23, p=0.863)	+0.187 (24/26, p=0.674)
ref_add: ep021 − ep018	-0.094 (20/30, p=0.149)	-0.012 (16/34, p=0.001)	+0.004 (21/29, p=0.105)	-0.176 (32/18, p=0.063)	-0.097 (21/29, p=0.592)

Verdict (100 clips): stop at ep014. ep015 improves on no metric. SSIM drops significantly in both directions (removal 20/80 clips better, addition 19/81, both p<0.001). Addition also gets worse on PSNR (p=0.015) and LPIPS (26/74, p<0.001). FVD and FAD rise in both cells (removal FVD 616→883, FAD 0.53→1.03). Audio KL/LSD changes are not significant. mainline ep021 is not better than ep018 either (50 clips: SSIM worse, p=0.036 removal / 0.001 addition).

Removal: mainline ep018 vs refwarm ep014 / ep015 on 100 val clips (no first frame) — 50 examples →

checkpoint	PSNR↑	SSIM↑	LPIPS↓	FVD↓	FAD↓	KL↓	LSD↓
mainline ep018	20.39	0.670	0.290	560	0.61	1.223	13.98
refwarm ep014	19.27	0.655	0.313	616	0.53	1.300	14.08
refwarm ep015	19.15	0.627	0.317	883	1.03	1.401	14.76

Paired per-clip change (first-named better / worse, Wilcoxon p).

pair	ΔPSNR	ΔSSIM	ΔLPIPS	ΔKL	ΔLSD
refwarm ep014 − mainline ep018	-1.124 (26/74, p=0.000)	-0.015 (28/72, p=0.000)	+0.023 (30/70, p=0.000)	+0.077 (41/59, p=0.207)	+0.100 (40/60, p=0.403)
refwarm ep015 − mainline ep018	-1.245 (22/78, p=0.000)	-0.043 (15/85, p=0.000)	+0.027 (24/76, p=0.000)	+0.178 (45/55, p=0.327)	+0.777 (45/55, p=0.193)
refwarm ep015 − refwarm ep014	-0.121 (39/61, p=0.080)	-0.028 (20/80, p=0.000)	+0.004 (41/59, p=0.051)	+0.101 (47/53, p=0.741)	+0.677 (51/49, p=0.575)

Verdict (removal, 100 val clips): mainline ep018 is better than both refwarm checkpoints. Against ep014, ep018 wins on PSNR (+1.12 dB, better on 74/100 clips), SSIM (72/100) and LPIPS (70/100), all p<0.001, and has lower FVD (560 vs 616). Audio is a tie: KL and LSD differences are not significant, and FAD is 0.61 vs 0.53, a set-level number that is noisy at n=100. ep015 is worse still (SSIM 85/100 clips in ep018's favour). refwarm's only removal edge is on the test50 DeSync, which does not separate methods in removal (the GT itself scores ~0.77).

AVI-Edit (1st frame)

ROSE + ZEUS