Data Sources & Updates

Comprehensive information about data sources, loading processes, and update procedures

Data Sources

Hebrew Text Files
  • tanakh_pointed_hebrew.txt - Hebrew text with vowel points Active
  • tanakh_unpointed_hebrew.txt - Hebrew text without vowel points Active
  • tanakh_flattened.txt - Continuous Hebrew text Active
English & Metadata
  • torah_english.txt - English Torah translation Active
  • tanakh_metadata.json - Book metadata and structure Active
  • tanakh_statistics.txt - Statistical data Optional

Data Loading Process

How Data is Loaded
  1. Application Startup: The load_data() function runs when the Flask app starts
  2. File Reading: All text files and JSON metadata are read into memory
  3. Data Structure: Content is organized in a global data dictionary
  4. Memory Storage: All data remains in RAM for fast access during user requests

How to Update Data

Manual Update Process
  1. Stop the Application: Terminate the Flask server
  2. Replace Files: Update the relevant text or JSON files
  3. Restart Application: Run python web_app.py again
  4. Verification: Check that new data appears correctly
File Locations

All data files are located in:

D:\Playground\

Ensure file names and formats match exactly when updating

Data Structure Overview 

data = {
    'metadata': {          # Book information from tanakh_metadata.json
        'Genesis': {...},
        'Exodus': {...},
        ...
    },
    'hebrew_pointed': "...",    # Content from tanakh_pointed_hebrew.txt
    'hebrew_unpointed': "...",  # Content from tanakh_unpointed_hebrew.txt
    'hebrew_flattened': "...",  # Content from tanakh_flattened.txt
    'english_torah': "...",     # Content from torah_english.txt
    'statistics': "...",        # Content from tanakh_statistics.txt
    'connections': [...]         # Parsed connection data
}
Important Notes
  • No Hot Reload: Changes to data files require application restart
  • Memory Usage: All text is loaded into RAM - monitor memory usage with large files
  • File Encoding: Ensure Hebrew text files use UTF-8 encoding
  • Backup: Always backup original files before making changes