File size: 41,559 Bytes
d79c4a9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
import os
import re
import torch
import warnings
import numpy as np
import gradio as gr
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig
)
from sentence_transformers import SentenceTransformer
from typing import List, Dict, Optional
import time
from datetime import datetime

# Suppress warnings
warnings.filterwarnings('ignore')

class BioGPTMedicalChatbot:
    def __init__(self):
        """Initialize BioGPT chatbot for Gradio deployment"""
        print("πŸ₯ Initializing BioGPT Pediatric Pulmonology Chatbot...")
        
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.use_8bit = torch.cuda.is_available()
        
        print(f"πŸ–₯️ Using device: {self.device}")
        
        # Setup components
        self.setup_embeddings()
        self.setup_biogpt()
        
        # Knowledge base and conversation tracking
        self.knowledge_chunks = []
        self.conversation_history = []
        
        # Load default medical knowledge first
        self.load_default_medical_knowledge()
        
        # Try to load your specific pediatric pulmonology data
        self.load_pediatric_pulmonology_data()
        
        print("βœ… BioGPT Pediatric Pulmonology Chatbot ready!")

    def load_pediatric_pulmonology_data(self):
        """Auto-load pediatric pulmonology data from uploaded file"""
        pulmonology_files = [
            'Pediatric_cleaned.txt',
            'pediatric_cleaned.txt', 
            'Pediatric_Cleaned.txt',
            'pediatric_pulmonology.txt',
            'pulmonology_data.txt'
        ]
        
        for filename in pulmonology_files:
            if os.path.exists(filename):
                print(f"πŸ“– Found pediatric pulmonology data: {filename}")
                try:
                    success = self.load_medical_data(filename)
                    if success:
                        print(f"βœ… Successfully loaded {filename} with pulmonology data!")
                        print(f"πŸ“Š Total knowledge chunks: {len(self.knowledge_chunks)}")
                        return True
                except Exception as e:
                    print(f"⚠️ Failed to load {filename}: {e}")
                    continue
        
        print("⚠️ No pediatric pulmonology data file found.")
        print("   Expected files: Pediatric_cleaned.txt")
        print("   Using default pediatric knowledge only.")
        print(f"πŸ“Š Current knowledge chunks: {len(self.knowledge_chunks)}")
        return False

    def load_medical_data(self, file_path: str):
        """Load and process medical data from text file"""
        print(f"πŸ“– Loading medical data from {file_path}...")

        try:
            with open(file_path, 'r', encoding='utf-8') as f:
                text = f.read()
            print(f"πŸ“„ File loaded: {len(text):,} characters")
        except FileNotFoundError:
            print(f"❌ File {file_path} not found!")
            return False
        except Exception as e:
            print(f"❌ Error reading file: {e}")
            return False

        # Create chunks optimized for medical content
        print("πŸ“ Creating pediatric pulmonology chunks...")
        new_chunks = self.create_medical_chunks_from_text(text)
        print(f"πŸ“‹ Created {len(new_chunks)} new medical chunks from file")

        # Add to existing knowledge chunks (don't replace, append)
        starting_id = len(self.knowledge_chunks)
        for i, chunk in enumerate(new_chunks):
            chunk['id'] = starting_id + i
            chunk['source'] = 'pediatric_pulmonology_file'
            
        self.knowledge_chunks.extend(new_chunks)
        
        print(f"βœ… Medical data loaded successfully!")
        print(f"πŸ“Š Total knowledge chunks: {len(self.knowledge_chunks)}")
        return True

    def create_medical_chunks_from_text(self, text: str, chunk_size: int = 400) -> List[Dict]:
        """Create medically-optimized text chunks from uploaded file"""
        chunks = []

        # Clean the text first - remove XML/HTML tags and formatting artifacts
        cleaned_text = self.clean_medical_text(text)

        # Split by medical sections first
        medical_sections = self.split_by_medical_sections(cleaned_text)

        for section in medical_sections:
            if len(section.split()) > chunk_size:
                # Split large sections by sentences
                sentences = re.split(r'[.!?]+', section)
                current_chunk = ""

                for sentence in sentences:
                    sentence = sentence.strip()
                    if not sentence:
                        continue

                    if len(current_chunk.split()) + len(sentence.split()) < chunk_size:
                        current_chunk += sentence + ". "
                    else:
                        if current_chunk.strip():
                            chunks.append({
                                'text': current_chunk.strip(),
                                'medical_focus': self.identify_medical_focus(current_chunk)
                            })
                        current_chunk = sentence + ". "

                if current_chunk.strip():
                    chunks.append({
                        'text': current_chunk.strip(),
                        'medical_focus': self.identify_medical_focus(current_chunk)
                    })
            else:
                if section.strip():
                    chunks.append({
                        'text': section.strip(),
                        'medical_focus': self.identify_medical_focus(section)
                    })

        return chunks

    def clean_medical_text(self, text: str) -> str:
        """Clean medical text from formatting artifacts and XML tags"""
        # Remove XML/HTML tags like </FREETEXT>, </ABSTRACT>, <SECTION>, etc.
        text = re.sub(r'<[^>]+>', '', text)
        
        # Remove common document formatting artifacts
        text = re.sub(r'</?\s*FREETEXT\s*>', '', text, flags=re.IGNORECASE)
        text = re.sub(r'</?\s*ABSTRACT\s*>', '', text, flags=re.IGNORECASE)
        text = re.sub(r'</?\s*SECTION\s*>', '', text, flags=re.IGNORECASE)
        text = re.sub(r'</?\s*TITLE\s*>', '', text, flags=re.IGNORECASE)
        
        # Remove excessive whitespace and newlines
        text = re.sub(r'\n\s*\n\s*\n+', '\n\n', text)
        text = re.sub(r'\s+', ' ', text)
        
        # Remove special characters that might be formatting artifacts
        text = re.sub(r'[^\w\s.,;:!?()\-\'/"]', ' ', text)
        
        # Clean up multiple spaces
        text = re.sub(r'\s+', ' ', text).strip()
        
        return text

    def split_by_medical_sections(self, text: str) -> List[str]:
        """Split text by medical sections"""
        # Look for medical section headers
        section_patterns = [
            r'\n\s*(?:SYMPTOMS?|TREATMENT|DIAGNOSIS|CAUSES?|PREVENTION|MANAGEMENT).*?\n',
            r'\n\s*\d+\.\s+',  # Numbered sections
            r'\n\n+'  # Paragraph breaks
        ]

        sections = [text]
        for pattern in section_patterns:
            new_sections = []
            for section in sections:
                splits = re.split(pattern, section, flags=re.IGNORECASE)
                new_sections.extend([s.strip() for s in splits if len(s.strip()) > 100])
            sections = new_sections

        return sections

    def identify_medical_focus(self, text: str) -> str:
        """Identify the medical focus of a text chunk with pulmonology emphasis"""
        text_lower = text.lower()

        # Enhanced medical categories with pulmonology focus
        categories = {
            'pediatric_pulmonology': [
                'asthma', 'pneumonia', 'bronchiolitis', 'croup', 'respiratory', 'lung', 'airway', 
                'breathing', 'cough', 'wheeze', 'stridor', 'pneumothorax', 'pleural', 'ventilator',
                'oxygen', 'respiratory distress', 'bronchitis', 'pulmonary', 'chest', 'inhaler'
            ],
            'pediatric_symptoms': ['fever', 'rash', 'vomiting', 'diarrhea', 'pain'],
            'treatments': ['treatment', 'therapy', 'medication', 'antibiotics', 'steroid'],
            'diagnosis': ['diagnosis', 'diagnostic', 'symptoms', 'signs', 'test'],
            'emergency': ['emergency', 'urgent', 'serious', 'hospital', 'icu'],
            'prevention': ['prevention', 'vaccine', 'immunization', 'avoid']
        }

        for category, keywords in categories.items():
            if any(keyword in text_lower for keyword in keywords):
                return category

        return 'general_medical'

    def setup_embeddings(self):
        """Setup medical embeddings"""
        try:
            print("πŸ”§ Loading embeddings...")
            self.embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
            self.use_embeddings = True
            print("βœ… Embeddings loaded successfully")
        except Exception as e:
            print(f"⚠️ Embeddings failed: {e}")
            self.embedding_model = None
            self.use_embeddings = False

    def setup_biogpt(self):
        """Setup BioGPT model with fallback"""
        print("🧠 Loading BioGPT model...")
        
        # Try BioGPT first, then fallback to smaller model
        models_to_try = [
            "microsoft/BioGPT-Large",
            "microsoft/BioGPT", 
            "microsoft/DialoGPT-medium"
        ]
        
        for model_name in models_to_try:
            try:
                print(f"   Trying {model_name}...")
                
                # Setup quantization for memory efficiency
                quantization_config = None
                if self.use_8bit and "BioGPT" in model_name:
                    quantization_config = BitsAndBytesConfig(
                        load_in_8bit=True,
                        llm_int8_threshold=6.0,
                    )

                # Load tokenizer
                self.tokenizer = AutoTokenizer.from_pretrained(model_name)
                if self.tokenizer.pad_token is None:
                    self.tokenizer.pad_token = self.tokenizer.eos_token

                # Load model
                self.model = AutoModelForCausalLM.from_pretrained(
                    model_name,
                    quantization_config=quantization_config,
                    torch_dtype=torch.float16 if self.device == "cuda" else torch.float32,
                    device_map="auto" if self.device == "cuda" and quantization_config else None,
                    trust_remote_code=True
                )
                
                if self.device == "cuda" and quantization_config is None:
                    self.model = self.model.to(self.device)
                
                print(f"βœ… Successfully loaded {model_name}!")
                self.model_name = model_name
                return
                
            except Exception as e:
                print(f"❌ Failed to load {model_name}: {e}")
                continue
        
        # If all models fail
        print("❌ All models failed to load")
        self.model = None
        self.tokenizer = None
        self.model_name = "None"

    def load_default_medical_knowledge(self):
        """Load comprehensive default medical knowledge base"""
        default_knowledge = [
            {
                'id': 0,
                'text': "Fever in children is commonly caused by viral infections (most common), bacterial infections, immunizations, teething in infants, or overdressing. Normal body temperature ranges from 97Β°F to 100.4Β°F (36.1Β°C to 38Β°C). A fever is generally considered when oral temperature exceeds 100.4Β°F (38Β°C). Most fevers are not dangerous and help the body fight infection. Treatment includes rest, fluids, and fever reducers like acetaminophen or ibuprofen for comfort.",
                'medical_focus': 'pediatric_symptoms',
                'source': 'default_knowledge'
            },
            {
                'id': 1,
                'text': "Dehydration in infants and children can occur rapidly, especially during illness with vomiting or diarrhea. Warning signs include: dry mouth and tongue, decreased urination, lethargy or irritability, sunken eyes, and in infants under 12 months, sunken fontanelle (soft spot). Mild dehydration can be treated with oral rehydration solutions or clear fluids. Severe dehydration requires immediate medical attention.",
                'medical_focus': 'pediatric_symptoms',
                'source': 'default_knowledge'
            },
            {
                'id': 2,
                'text': "Common cold symptoms in children include runny or stuffy nose, cough, low-grade fever, sneezing, and general fussiness. Most colds are viral and resolve within 7-10 days without specific treatment. Treatment focuses on comfort measures: rest, adequate fluids, humidified air, and saline nasal drops for congestion. Antibiotics are not effective against viral colds.",
                'medical_focus': 'pediatric_symptoms',
                'source': 'default_knowledge'
            },
            {
                'id': 3,
                'text': "Emergency warning signs in children requiring immediate medical attention include: severe difficulty breathing, persistent high fever over 104Β°F (40Β°C), signs of severe dehydration, persistent vomiting preventing fluid intake, severe headache with neck stiffness, altered consciousness or extreme lethargy, severe abdominal pain, or any concerning change in behavior. When in doubt, seek medical care.",
                'medical_focus': 'emergency',
                'source': 'default_knowledge'
            },
            {
                'id': 4,
                'text': "Childhood vaccination schedules protect against serious diseases including measles, mumps, rubella, polio, hepatitis B, Haemophilus influenzae, pneumococcal disease, and others. Vaccines are rigorously tested for safety and effectiveness. Side effects are typically mild, such as low-grade fever or soreness at injection site. Following recommended vaccination schedules protects individual children and the community.",
                'medical_focus': 'prevention',
                'source': 'default_knowledge'
            },
            {
                'id': 5,
                'text': "Persistent cough in children can be caused by viral upper respiratory infections, asthma, allergies, bacterial infections, or irritants. Most coughs from colds resolve within 2-3 weeks. Seek medical evaluation for coughs lasting more than 3 weeks, coughs with blood, difficulty breathing, or coughs severely interfering with sleep. Treatment depends on the underlying cause.",
                'medical_focus': 'pediatric_symptoms',
                'source': 'default_knowledge'
            },
            # Adding more default knowledge chunks for comprehensive coverage...
            {
                'id': 6,
                'text': "Asthma in children is a chronic respiratory condition affecting the airways, causing them to become inflamed, narrow, and produce excess mucus. Common triggers include viral infections, allergens (dust mites, pollen, pet dander), irritants (smoke, strong odors), cold air, and exercise. Symptoms include wheezing, cough (especially at night), shortness of breath, and chest tightness. Management includes avoiding triggers, using prescribed inhalers, and having an asthma action plan.",
                'medical_focus': 'pediatric_pulmonology',
                'source': 'default_knowledge'
            },
            {
                'id': 7,
                'text': "Bronchiolitis is a common respiratory infection in infants and young children, typically caused by respiratory syncytial virus (RSV). It affects the small airways (bronchioles) in the lungs, causing inflammation and mucus buildup. Symptoms include runny nose, cough, low-grade fever, and difficulty breathing. Most cases are mild and resolve with supportive care, but severe cases may require hospitalization for oxygen support.",
                'medical_focus': 'pediatric_pulmonology',
                'source': 'default_knowledge'
            }
        ]
        
        self.knowledge_chunks = default_knowledge
        print(f"πŸ“š Loaded {len(default_knowledge)} default medical knowledge chunks")

    def retrieve_medical_context(self, query: str, n_results: int = 3) -> List[str]:
        """Retrieve relevant medical context using improved keyword search with pulmonology priority"""
        if not self.knowledge_chunks:
            return []
            
        query_lower = query.lower()
        query_words = set(query_lower.split())
        chunk_scores = []

        # Enhanced medical keyword mapping with pulmonology emphasis
        medical_keywords = {
            'pulmonology': ['asthma', 'pneumonia', 'bronchiolitis', 'croup', 'respiratory', 'lung', 'airway', 'breathing', 'cough', 'wheeze', 'stridor'],
            'fever': ['fever', 'temperature', 'hot', 'warm', 'burning'],
            'stomach': ['stomach', 'abdominal', 'belly', 'tummy', 'pain', 'ache'],
            'rash': ['rash', 'skin', 'red', 'spots', 'bumps', 'itchy'],
            'vomiting': ['vomit', 'vomiting', 'throw up', 'sick', 'nausea'],
            'diarrhea': ['diarrhea', 'loose', 'stool', 'bowel', 'poop'],
            'dehydration': ['dehydration', 'dehydrated', 'fluids', 'water', 'thirsty'],
            'emergency': ['emergency', 'urgent', 'serious', 'severe', 'hospital', 'doctor']
        }

        # Expand query with related medical terms
        expanded_query_words = set(query_words)
        for medical_term, synonyms in medical_keywords.items():
            if any(word in query_lower for word in synonyms):
                expanded_query_words.update(synonyms)

        for chunk_info in self.knowledge_chunks:
            chunk_text = chunk_info['text'].lower()
            
            # Calculate relevance score with expanded terms
            word_overlap = sum(1 for word in expanded_query_words if word in chunk_text)
            base_score = word_overlap / len(expanded_query_words) if expanded_query_words else 0

            # Strong boost for pulmonology content
            medical_boost = 0
            medical_focus = chunk_info.get('medical_focus', '')
            source = chunk_info.get('source', '')
            
            if medical_focus == 'pediatric_pulmonology':
                medical_boost = 0.8  # Highest priority for pulmonology
            elif source == 'pediatric_pulmonology_file':
                medical_boost = 0.7  # High priority for your uploaded data
            elif medical_focus == 'emergency':
                medical_boost = 0.4
            elif medical_focus in ['treatments', 'diagnosis']:
                medical_boost = 0.3
            elif medical_focus == 'pediatric_symptoms':
                medical_boost = 0.5

            final_score = base_score + medical_boost

            if final_score > 0:
                chunk_scores.append((final_score, chunk_info['text']))

        # Return top matches - prioritize pulmonology content
        chunk_scores.sort(reverse=True)
        results = [chunk for _, chunk in chunk_scores[:n_results]]
        
        # If no good matches, return some default medical chunks
        if not results:
            results = [chunk['text'] for chunk in self.knowledge_chunks[:2]]
            
        return results

    def generate_biogpt_response(self, context: str, query: str) -> str:
        """Generate medical response using BioGPT with improved prompting"""
        if not self.model or not self.tokenizer:
            return "Medical model not available. Please try again later."

        try:
            # Create a more focused medical prompt for better responses
            prompt = f"""You are a pediatric medical expert. Based on the following medical information, provide a clear, helpful answer to the patient's question.

Medical Context: {context[:600]}

Patient Question: {query}

Your response should be a direct medical answer. Do not ask follow-up questions. Just provide the best possible explanation or advice based on the context."""

            # Tokenize input
            inputs = self.tokenizer(
                prompt,
                return_tensors="pt",
                truncation=True,
                max_length=900  # Reduced to leave more room for generation
            )

            # Move inputs to device
            if self.device == "cuda":
                inputs = {k: v.to(self.device) for k, v in inputs.items()}

            # Generate response with improved parameters
            with torch.no_grad():
                outputs = self.model.generate(
                    **inputs,
                    max_new_tokens=200,  # Increased for more detailed responses
                    do_sample=True,
                    temperature=0.6,  # Reduced for more focused responses
                    top_p=0.85,  # Slightly reduced for better quality
                    top_k=40,  # Added top-k for better control
                    pad_token_id=self.tokenizer.eos_token_id,
                    repetition_penalty=1.15,  # Increased to reduce repetition
                    no_repeat_ngram_size=3  # Prevent 3-gram repetition
                )

            # Decode response
            full_response = self.tokenizer.decode(outputs[0], skip_special_tokens=True)

            # Extract generated part more carefully
            if "Medical Response" in full_response:
                generated_response = full_response.split("Medical Response")[-1].strip()
                # Remove any remaining prompt artifacts
                generated_response = re.sub(r'^\W*\([^)]*\)\W*', '', generated_response)
                generated_response = generated_response.lstrip(':').strip()
            else:
                generated_response = full_response[len(prompt):].strip()

            # Clean and improve response
            cleaned_response = self.clean_and_improve_medical_response(generated_response)

            return cleaned_response

        except Exception as e:
            print(f"⚠️ BioGPT generation failed: {e}")
            return self.fallback_response(context, query)

    def clean_and_improve_medical_response(self, response: str) -> str:
        """Enhanced cleaning and improvement of medical response"""
        # Remove any remaining XML/formatting artifacts
        response = re.sub(r'<[^>]+>', '', response)
        response = re.sub(r'</?\s*\w+\s*>', '', response)
        
        # Remove incomplete or malformed sentences at the end
        sentences = re.split(r'[.!?]+', response)
        clean_sentences = []

        for sentence in sentences:
            sentence = sentence.strip()
            # Keep sentences that are substantial and don't end with incomplete words
            if (len(sentence) > 15 and 
                not sentence.endswith(('and', 'or', 'but', 'however', 'the', 'a', 'an', 'in', 'on', 'at')) and
                not re.search(r'\b\w{1,2}$', sentence)):  
                clean_sentences.append(sentence)
            if len(clean_sentences) >= 4:  # Limit to 4 good sentences
                break

        if clean_sentences:
            cleaned = '. '.join(clean_sentences) + '.'
            # Ensure it starts with a capital letter
            if cleaned and cleaned[0].islower():
                cleaned = cleaned[0].upper() + cleaned[1:]
        else:
            # If no good sentences, try to extract the first meaningful part
            cleaned = response[:300].strip()
            if not cleaned.endswith('.'):
                cleaned += '.'

        # Final cleanup
        cleaned = re.sub(r'\s+', ' ', cleaned).strip()
        
        return cleaned
    def clean_medical_response(self, response: str) -> str:
        """Clean medical response"""
        sentences = re.split(r'[.!?]+', response)
        clean_sentences = []

        for sentence in sentences:
            sentence = sentence.strip()
            if len(sentence) > 10:
                clean_sentences.append(sentence)
            if len(clean_sentences) >= 3:
                break

        if clean_sentences:
            cleaned = '. '.join(clean_sentences) + '.'
        else:
            cleaned = response[:200] + '...' if len(response) > 200 else response

        return cleaned

    def fallback_response(self, context: str, query: str) -> str:
        """Fallback response when model fails"""
        sentences = [s.strip() for s in context.split('.') if len(s.strip()) > 20]
        
        if sentences:
            response = sentences[0] + '.'
            if len(sentences) > 1:
                response += ' ' + sentences[1] + '.'
        else:
            response = context[:300] + '...'
            
        return response

    def handle_conversational_interactions(self, query: str) -> Optional[str]:
        """Handle conversational interactions"""
        query_lower = query.lower().strip()

        # Greeting patterns
        exact_greetings = [
            'hello', 'hi', 'hey', 'good morning', 'good afternoon', 
            'good evening', 'how are you', 'how are you doing'
        ]
        
        if query_lower in exact_greetings:
            return "πŸ‘‹ Hello! I'm BioGPT, your AI medical assistant specialized in pediatric pulmonology. I provide evidence-based medical information about children's respiratory health. What can I help you with today?"

        # Thanks patterns
        thanks_only = ['thank you', 'thanks', 'thank you so much', 'thanks a lot']
        if query_lower in thanks_only:
            return "πŸ™ You're welcome! I'm glad I could provide helpful pediatric pulmonology information. Remember to always consult healthcare providers for personalized advice. Feel free to ask more questions!"

        # Help patterns
        help_only = ['help', 'what can you do', 'what are you', 'who are you']
        if query_lower in help_only:
            return """πŸ€– **About BioGPT Pediatric Pulmonology Assistant**

I'm an AI medical assistant powered by BioGPT, specialized in pediatric pulmonology and respiratory medicine. I can help with:

🫁 **Pediatric Pulmonology:**
β€’ Asthma, bronchiolitis, pneumonia, croup
β€’ Respiratory symptoms and breathing difficulties
β€’ Treatment guidance and management
β€’ When to seek medical care

⚠️ **Important:** I provide educational information only. Always consult healthcare professionals for medical decisions."""

        return None

    def chat_interface(self, message: str, history: List[List[str]]) -> str:
        """Main chat interface for Gradio"""
        if not message.strip():
            return "Hello! I'm BioGPT, your pediatric pulmonology AI assistant. How can I help you with children's respiratory health today?"

        print(f"πŸ” Processing query: '{message}'")

        # Handle conversational interactions
        conversational_response = self.handle_conversational_interactions(message)
        if conversational_response:
            print("   Handled as conversational")
            return conversational_response

        print("   Processing as medical query")

        # Process as medical query
        context = self.retrieve_medical_context(message)
        
        if not context:
            return f"""🩺 **Medical Query:** {message}

⚠️ I don't have specific information about this topic in my current medical database. However, I recommend:

1. **Consult Healthcare Provider**: For personalized medical advice
2. **Emergency Signs**: If symptoms are severe, seek immediate care
3. **Pediatric Specialist**: For specialized concerns

**For urgent medical concerns, contact your healthcare provider or emergency services.**

πŸ’‘ **Try asking about**: asthma, breathing difficulties, cough, pneumonia, or other respiratory symptoms."""

        # Generate medical response
        main_context = '\n\n'.join(context)
        response = self.generate_biogpt_response(main_context, message)

        # Format as medical response
        final_response = f"🩺 **BioGPT Medical Assistant:** {response}\n\n⚠️ **Important:** This information is for educational purposes only. Always consult qualified healthcare professionals for medical diagnosis, treatment, and personalized advice."

        return final_response

    def get_knowledge_stats(self) -> Dict:
        """Get statistics about loaded knowledge"""
        if not self.knowledge_chunks:
            return {"total_chunks": 0}
        
        stats = {
            "total_chunks": len(self.knowledge_chunks),
            "default_knowledge": len([c for c in self.knowledge_chunks if c.get('source') == 'default_knowledge']),
            "pulmonology_file_data": len([c for c in self.knowledge_chunks if c.get('source') == 'pediatric_pulmonology_file']),
            "pulmonology_focused": len([c for c in self.knowledge_chunks if c.get('medical_focus') == 'pediatric_pulmonology']),
            "model_used": getattr(self, 'model_name', 'Unknown')
        }
        return stats

# Test function
def test_chatbot_responses():
    """Test the chatbot with various queries"""
    print("\nπŸ§ͺ Testing BioGPT Pediatric Pulmonology Chatbot...")
    print("=" * 50)
    
    test_queries = [
        "hello",
        "what is asthma in children",
        "my child has breathing difficulties",
        "help",
        "treatment for pediatric pneumonia",
        "thank you"
    ]
    
    for query in test_queries:
        print(f"\nπŸ” Query: '{query}'")
        response = chatbot.chat_interface(query, [])
        response_type = 'CONVERSATIONAL' if any(word in response for word in ['Hello!', 'welcome!', 'About BioGPT']) else 'MEDICAL'
        print(f"πŸ€– Response type: {response_type}")
        print(f"πŸ“ Response: {response[:100]}...")
        print("-" * 30)

# Initialize the chatbot globally
print("πŸš€ Initializing BioGPT Pediatric Pulmonology Chatbot...")
chatbot = BioGPTMedicalChatbot()

# Show knowledge statistics
print("\nπŸ“Š Knowledge Base Statistics:")
stats = chatbot.get_knowledge_stats()
for key, value in stats.items():
    print(f"   {key}: {value}")

# Run tests
test_chatbot_responses()

def create_gradio_interface():
    """Create Gradio chat interface"""
    
    # Get current knowledge stats for display
    stats = chatbot.get_knowledge_stats()
    
    # Custom CSS for medical theme
    css = """
    .gradio-container {
        background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
    }
    .chat-message {
        background-color: #f8f9fa;
        border-radius: 10px;
        padding: 10px;
        margin: 5px;
    }
    """
    
    with gr.Blocks(
        css=css,
        title="BioGPT Pediatric Pulmonology Assistant",
        theme=gr.themes.Soft()
    ) as demo:
        
        # Header
        gr.HTML(f"""
        <div style="text-align: center; padding: 20px; background: linear-gradient(90deg, #667eea, #764ba2); color: white; border-radius: 10px; margin-bottom: 20px;">
            <h1>🫁 BioGPT Pediatric Pulmonology Assistant</h1>
            <p>Specialized AI Medical Chatbot for Children's Respiratory Health</p>
            <p><strong>Powered by BioGPT | {stats['total_chunks']} Medical Knowledge Chunks Loaded</strong></p>
            <p><small>Model: {stats['model_used']} | Pulmonology Data: {stats['pulmonology_file_data']} chunks</small></p>
        </div>
        """)
        
        # Important disclaimer
        gr.HTML("""
        <div style="background-color: #fff3cd; border: 1px solid #ffeaa7; border-radius: 8px; padding: 15px; margin-bottom: 20px;">
            <h3 style="color: #856404; margin-top: 0;">⚠️ Medical Disclaimer</h3>
            <p style="color: #856404; margin-bottom: 0;">
                This AI provides educational pediatric pulmonology information only and is NOT a substitute for professional medical advice, 
                diagnosis, or treatment. Always consult qualified healthcare providers for medical decisions.
                <strong>In case of respiratory emergency, call emergency services immediately.</strong>
            </p>
        </div>
        """)
        
        # Chat interface
        chatbot_interface = gr.ChatInterface(
            fn=chatbot.chat_interface,
            title="πŸ’¬ Chat with BioGPT Pulmonology Assistant",
            description="Ask me about pediatric respiratory health, asthma, breathing difficulties, and pulmonology treatments.",
            examples=[
                "What is asthma in children?",
                "My child has a persistent cough, what should I do?",
                "How is bronchiolitis treated in infants?",
                "When should I be worried about my child's breathing?",
                "What are the signs of pneumonia in children?",
                "How can I prevent respiratory infections?"
            ],
            retry_btn=None,
            undo_btn=None,
            clear_btn="πŸ—‘οΈ Clear Chat",
            submit_btn="🫁 Ask BioGPT",
            chatbot=gr.Chatbot(
                height=500,
                placeholder="<div style='text-align: center; color: #666;'>Start a conversation with BioGPT Pediatric Pulmonology Assistant</div>",
                show_copy_button=True,
                bubble_full_width=False
            )
        )
        
        # Information tabs
        with gr.Tabs():
            with gr.Tab("ℹ️ About"):
                gr.Markdown(f"""
                ## About BioGPT Pediatric Pulmonology Assistant
                
                This AI assistant is powered by **BioGPT**, specialized for pediatric pulmonology and respiratory medicine.
                
                ### 🎯 Current Knowledge Base:
                - **Total Chunks**: {stats['total_chunks']}
                - **Default Medical Knowledge**: {stats['default_knowledge']} chunks
                - **Pulmonology File Data**: {stats['pulmonology_file_data']} chunks  
                - **Pulmonology Focus**: {stats['pulmonology_focused']} chunks
                - **Model**: {stats['model_used']}
                
                ### 🫁 Specializations:
                - **Pediatric Asthma**: Diagnosis, treatment, management
                - **Respiratory Infections**: Pneumonia, bronchiolitis, croup
                - **Breathing Difficulties**: Assessment and guidance
                - **Chronic Respiratory Conditions**: Long-term management
                - **Emergency Respiratory Care**: When to seek immediate help
                
                ### πŸ”§ Technical Features:
                - **Model**: Microsoft BioGPT (Medical AI)
                - **Auto-Loading**: Automatically loads your pulmonology data file
                - **Smart Retrieval**: Prioritizes pulmonology content
                - **Fallback System**: Ensures reliability across different environments
                
                ### πŸ“± How to Use:
                1. Type your pediatric respiratory question
                2. Be specific about symptoms or conditions
                3. Ask about treatments, diagnosis, or management
                4. Request guidance on when to seek care
                """)
            
            with gr.Tab("🫁 Pulmonology Topics"):
                gr.Markdown("""
                ## Pediatric Pulmonology Coverage
                
                ### πŸ”΄ Common Respiratory Conditions:
                - **Asthma**: Triggers, symptoms, management, action plans
                - **Bronchiolitis**: RSV, treatment, when to hospitalize
                - **Pneumonia**: Bacterial vs viral, antibiotics, recovery
                - **Croup**: Barking cough, stridor, home treatment
                - **Bronchitis**: Acute vs chronic, treatment approaches
                
                ### 🟑 Respiratory Symptoms:
                - **Cough**: Persistent, productive, dry, nocturnal
                - **Wheezing**: Causes, assessment, treatment
                - **Shortness of Breath**: Evaluation and management
                - **Chest Pain**: When concerning in children
                - **Stridor**: Upper airway obstruction signs
                
                ### 🟒 Diagnostic & Treatment:
                - **Pulmonary Function Tests**: When appropriate
                - **Imaging**: X-rays, CT scans for respiratory issues
                - **Medications**: Bronchodilators, steroids, antibiotics
                - **Oxygen Therapy**: Indications and monitoring
                - **Respiratory Support**: CPAP, ventilation considerations
                
                ### πŸ”΅ Prevention & Management:
                - **Trigger Avoidance**: Environmental controls
                - **Vaccination**: Respiratory disease prevention
                - **Exercise Guidelines**: For children with respiratory conditions
                - **School Management**: Asthma action plans, inhaler use
                """)
            
            with gr.Tab("⚠️ Emergency & Safety"):
                gr.Markdown("""
                ## Respiratory Emergency Guidance
                
                ### 🚨 CALL EMERGENCY SERVICES IMMEDIATELY:
                - **Severe Breathing Difficulty**: Cannot speak in full sentences
                - **Blue Lips or Fingernails**: Cyanosis indicating oxygen deprivation
                - **Severe Wheezing**: With significant distress
                - **Stridor at Rest**: High-pitched breathing sound
                - **Unconsciousness**: Related to breathing problems
                - **Severe Chest Retractions**: Pulling in around ribs/sternum
                
                ### πŸ₯ SEEK IMMEDIATE MEDICAL CARE:
                - **Persistent High Fever**: >104Β°F (40Β°C) with respiratory symptoms
                - **Worsening Symptoms**: Despite treatment
                - **Dehydration Signs**: With respiratory illness
                - **Significant Behavior Changes**: Extreme lethargy, irritability
                - **Inhaler Not Helping**: Asthma symptoms not responding
                
                ### πŸ“ž CONTACT HEALTHCARE PROVIDER:
                - **New Respiratory Symptoms**: Lasting more than a few days
                - **Chronic Cough**: Persisting beyond 2-3 weeks
                - **Asthma Questions**: About medications or management
                - **Fever with Cough**: Especially if productive
                - **Exercise Limitations**: Due to breathing difficulties
                
                ### 🏠 HOME MONITORING:
                - **Respiratory Rate**: Normal ranges by age
                - **Oxygen Saturation**: If pulse oximeter available
                - **Peak Flow**: For children with asthma
                - **Symptom Tracking**: Using asthma diaries or apps
                """)
                
            with gr.Tab("πŸ“ Data Information"):
                gr.Markdown(f"""
                ## Knowledge Base Status
                
                ### πŸ“Š Current Data Loaded:
                - **Total Medical Chunks**: {stats['total_chunks']}
                - **Default Knowledge**: {stats['default_knowledge']} chunks
                - **Your Pulmonology File**: {stats['pulmonology_file_data']} chunks
                - **Pulmonology Focused**: {stats['pulmonology_focused']} chunks
                - **AI Model**: {stats['model_used']}
                
                ### πŸ“ How Your Data Is Used:
                1. **Auto-Detection**: System automatically looks for 'Pediatric_cleaned.txt'
                2. **Smart Processing**: Breaks your file into medical chunks
                3. **Priority Ranking**: Pulmonology content gets highest priority
                4. **Context Retrieval**: Relevant chunks are used to answer questions
                
                ### πŸ“‚ Expected File Formats:
                - **Filename**: 'Pediatric_cleaned.txt' (case variations accepted)
                - **Content**: Plain text with medical information
                - **Structure**: Paragraphs, sections, or bullet points
                - **Focus**: Pediatric pulmonology and respiratory medicine
                
                ### πŸ”„ Upload Instructions:
                1. Go to your Hugging Face Space
                2. Click "Files" tab
                3. Upload your 'Pediatric_cleaned.txt' file
                4. Restart the Space to reload data
                
                {"βœ… **Status**: Pulmonology data file loaded successfully!" if stats['pulmonology_file_data'] > 0 else "⚠️ **Status**: No pulmonology data file detected. Upload 'Pediatric_cleaned.txt' to enhance responses."}
                """)
        
        # Footer
        gr.HTML("""
        <div style="text-align: center; padding: 20px; margin-top: 30px; border-top: 1px solid #ddd; color: #666;">
            <p>🫁 <strong>BioGPT Pediatric Pulmonology Assistant</strong> | Powered by Microsoft BioGPT</p>
            <p>Specialized in Children's Respiratory Health β€’ Always consult healthcare professionals</p>
        </div>
        """)
    
    return demo

# Create and launch the interface
demo = create_gradio_interface()

if __name__ == "__main__":
    # Launch the app
    demo.launch(
        server_name="0.0.0.0",
        server_port=7860,
        share=False
    )