【Adaptation】 TTA to Federated and Semi-supervised learning

- TTA to Federated and Semi-supervised learning # Test time adaptation - Notable properties 1. Out-of-distribution Generalization with target images in testing 2. **Self-training** anyway 3. Online update (optional) 4. Small parts of parameters (optional) 5. Fast optimization (empirical finding) 6. If update BN, the performance impact is due to $norm para > de-norm para$ (empirical finding) - Problems - Self-training for multi epochs → catastrophic failure → So then, Overfitting detector? - Learning what the model is already good at. → Even if any complex loss (ex, Consistency Training), no significant performance improvement by optimized de-norm. → Need to self-train what the model is bad at. - Local optimization - Small label efficiency compared to the large risk from on-device self-training. - Problems with my work 1. Domain division - Dividing into domains by weather and time was not proper since they have a small domain shift. (proof: [WILDS paper](https://arxiv.org/abs/2012.07421)) 2. Small margin performance improvement # Federated learning # Semi-supervised learning
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.8024412989616394, "perplexity": 8345.139706250962}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2022-40/segments/1664030337906.7/warc/CC-MAIN-20221007014029-20221007044029-00509.warc.gz"}