diff -ruN linux-2.4.24-uc0-snap1/Documentation/arm/FASS_ARM_Linux_Implementation linux-2.4.24-uc0-snap1-fass0/Documentation/arm/FASS_ARM_Linux_Implementation --- linux-2.4.24-uc0-snap1/Documentation/arm/FASS_ARM_Linux_Implementation 1970-01-01 10:00:00.000000000 +1000 +++ linux-2.4.24-uc0-snap1-fass0/Documentation/arm/FASS_ARM_Linux_Implementation 2004-03-29 09:02:55.000000000 +1000 @@ -0,0 +1,226 @@ +ARM Linux Fast Address-Space Switching (FASS) + + This is current (hopefully) as of patch diff-2.4.18-rmk4-fass4 it gives +an overview of the design and describes bits of the implementation, its +structure and status. + If you have any comments/corrections please let me know at + + + Cheers, Adam + +Basic Idea: + + Ok the basic Idea is as follows. Rather then switch page directories +(pg_dir) which requires the cache and TLB flushes a single pg_dir is used. I +call this global pg_dir the "Caching Page Directory" or CPD. It can be thought +of as a software TLB. + The CPD stores pg_dir entries from multiple user page directories as +well as the kernel's pg_dir entries. The pg_dir entries are tagged with ARM +domains which are assigned uniquely to user tasks as well as a small set being +reserved for tagging kernel mappings. + A context switch from one user process to another is carried out by +disabling the domain of the old process and enabling the domain of the new +process. The set of domains allocated to the kernel mappings are manipulated +as in the unmodified ARM Linux kernel. + Using this technique the caches and TLBs need only be flushed if one of +two events happen: + + 1) An entry in the CPD is replace by one owned by a different user +process, hence tagged with a different domain. + + 2) A domain is currently owned by one user process is reallocated to +another one. + + These issues both present a challenge to the design. 1) is a problem +since the Unix address-space layout is typically the same for every process. +2) is a problem since typically we have many more processes then ARM domains +which are fixed at 16 by the architecture. + These problems are solved as follows: + + 1) A dual approach is used here to handle the two types of memory +regions used in a user process: + + a) All memory regions with run time address bindings, ie those +mapped in by mmap() without the MAP_FIXED flag, are grouped into 1MB regions +(because pg_dir entries cover 1MB) that are already being used by the process. +If the new mapping doesn't fit in an existing 1MB region allocated to the user +process. A new one will be allocated, the newly allocated region will be one +that causes the least contention with other processes using that same 1MB +region, or the virtual address-space. Multiple processes can use the same 1MB +region safely because of the domain tags and CPD mechanism. 'Least contention' +can be defined as either least number of processes sharing a domain (likelihood +of a contention) or least number of previous conflicts (history of contention) +or some combination of the these. + + b) All memory regions with compile time address bindings, the +text/data/bss/etc which are mapped in by mmap() with the MAP_FIXED flag, are +as much as possible compiled to be within the first 32MB, ie below 0x02000000. +The mappings are then relocated using the ARM PID register into one of the 32MB +slots in the first 2GBs of the address-space. The relocation staggers 64 user +processes uniquely then PIDs well have to be shared. Again the overlapping is +safe as the CPD mechanism handles the contention. As there are more PIDs then +domains the number shouldn't be an issue. The stack is also set to the top of +the 32MB region, even though this could be placed anywhere, hence, staggered by +1a) it is placed in the PID relocation area because fork()ing will keep the +same stack pointer casuing a collision between the two processes. While this +will limit the size of brk() to be inside the 32MB area it does not limit +malloc() as it will use mmap() to allocate new heaps if brk() fails. If the +stack is simply staggered by 1a) the brk() grow past the PID relocation area as +the PID relocation is transparent to the user process and most of the kernel. +Only the CPD mechanism is aware of it. Actual allocation of PIDs still up in the +air. At the moment it simply round-robins through them. Smarter allocation +schemes might take into account the number of address-spaces using the PID and +allocate the one with the least usage or with the least busy address-spaces, +etc. To minimise collisions due to fork mappings flagged MAP_PRIVATE are as much +as possible allocated in the 32MB relocation area. + + 2) At the moment the approach is to revoke what causes the least ammount +of work, first we look for a clean domain to revoke. This means no cache flush +and only invalidating the CPD entries tagged with the victim domain. If no clean +domains are avaliable then a dirty one is used. In both cases the one with the +least number of entries in the CPD is used. This might not be the most optimal +way of doing it as tasks newly allocated a domain may be preempted first in +times of domain thrashing. + +Data Cache Aliasing: + + Aliases occur when the same physical frame is mapped to more then one +virtual address. This can happen within the same address-space or between +different address-spaces. It is only a problem for writable pages as the +different address lines will map to different cache lines. If one is modified +the other will not see the change so accesses to that virtual address will +access stale data. + The problem can be address in one of two ways: + + 1) Mark the shared mapping as uncachable. This slows down accesses to +the shared mappings but never requires cache flushing. ARM Linux currently uses +this method for shared mappings within the same address-space. + + 2) Only activate one mapping at a time. When another mapping is accessed +it will fault. At that stage the page in question is cleaned/flushed from the +DCache, the old mapping is deactivated, and the accessed mapping is activated. +faster accesses on the shared mappings but will cause slow downs when the +frame changes mappings. + + I plan to implement and benchmark both solutions. Or at least analise +it further, possibly have a flag to set which one is used. + + If the shared mapping uses the same virtual address in each +address-space no cache flushing or uncacheable setting is needed. Although +this will cause a CPD conflict anyway so no gain. Seperate domains for unique +sharing groups could solve this but is definately future work. + +Lazy Cache/TLB Flushing: + + All the cache and tlb flushing routines used to flush user mappings +make use of the following fact. If the CPD entry/entries mapping in the flush +range are not tagged with the domain tag of the address-space being flushed, +the caches cannot contain any data from that address-space and the TLBs cannot +contain any mappings for that address-space. This is used to avoid many cache +flushes. + Another bit of information can be used. When the caches/TLBs are flushed +all domains are marked as clean with respect to them. They remain clean until +the domain is activated. This way an address-space with CPD entries in the +flush region may still be clean if none of it's mappings have been touched. + This technique of lazy flushing could be used by normal ARM Linux using +the single user domain. + +Shared Domains: + + As well as using domains as a kind of address-space identifier (ASID) +tagging address-spaces uniquely they can be used to tagged shared mappings that +map to the same virtual addresses with the same size/permissions in each of the +sharing address-spaces. This is known as shared domains. To support this every +vm_area_struct (Linux mapping representation) is allocated a region, one is also +allocated to the address-space. Mappings which are private to the address-space +simply use the address-spaces region. Domains are now mapped to region's not +address spaces and when we switch to a new address space we enabled the domains, +if any, allocated to the regions accessable by the address-space. + +Implimentation Details: + + The implementation is spread around a LOT of the ARM specific files. +However FASS slides nicely into Linux with only ARM arch specific modifications. +The main part of FASS, the CPD mechanism, fits in the following way: + + * Entries are added into the CPD by ARM Data/Prefetch Aborts in +arch/arm/mm/fault-armv.c the abort handlers are modifed and the functions +do_cpd_fault() and do_domain_fault() are added. This functions go on to call +CPD management functions in arch/arm/mm/cpd.c + + * Entries are removed from the CPD by the functions in arch/arm/mm/cpd.c +this includes cpd_set_pmd() which updates the CPD to reflect changes in the +user pg_dir. CPD entries are also removed in cpd_load() due to a conflict +(replacement) and in domain_unallocate() due to domain preemption or the +address-space being torn down. + + * The cache/TLB flush routines in include/asm-arm/proc-armv/cache.h all +call an equivalent cpd_*() functions in arch/arm/mm/cpd.c to implement lazy +flushing of caches/TLBs. + + * ARM PID management code is in arch/arm/mm/pid.c and +include/asm-arm/proc-armv/pid.h + + * ARM domain management code is in arch/arm/mm/cpd.c , +include/asm-arm/proc-armv/cpd.h and include/asm-arm/proc-armv/domain.h + + * An address-space mmu_context is defined in include/asm-arm/mmu.h and +include/asm-arm/mmu_context.h defining a domain/PID allocated to the +address-space. + + * Some architecture independent code changes have been made to the code +that deals with vma's to add a new field 'vm_sharing_data' which points to the +region data structure in ARM and should be used for similar TLB sharing support +in other architectures like IA-64. 'vm_private_data' was not used as it is +unclear how this field is used by other Linux code such as drivers. + + All other changes should be clear from the patch. + +To do: + * arch/arm/mm/pid.c:pid_allocate() simply finds a PID with the lowest +number of address-spaces using it. There could possibly be a better way of doing +this. It also needs to decide when a ARM PID of 0, ie no VM relocation is used, +this might have to be decided by the caller of pid_allocate(). Binaries located +outside of the relocation area cause problems. + + * arch/arm/mm/fault-armv.c:update_mmu_cache() allocates region_structs +to shared mapping to support shared domains. However at the moment it doesn't +check the size or protection rights of the mapping allowing the max of each out +the process set to be accessable to all address-spaces sharing the mapping. This +is incorrect behaviour and needs to be delt fixed up some how. + + * arch/arm/mm/cpd.c:cpd_get_domain()/cpd_get_active_domain(), these +functions should be updated to use the address-spaces private region's domain +if the shared region has a count of one. This is to minimise domain thrashing, +it may not provide much benifit and is a little tricky to do correctly. + + * The code in cpd.c could probably be split up into seperate files, it +really should only have the cpd_load function and some helpers. + +Future Work: + + * Using sub-pages to deal with Cache Aliasing. This is another +experimental idea. If you look at the StrongARM-1 Core the cache is made up of +1KB colours, if you don't understand cache colouring ignore this. But basically +we use the (de)activation technique 2) for dealing with cache aliases except +we (de)activate the 1KB sub-pages and flush 1KB sub-pages at a time. This better +targets the flush and should reduce the overall cost. The implementation might +be tricky though. + + * Modifying the architectural independent scheduling goodness() function +to schedule around flushes, ie delay the flush as long as possible within a +scheduling epoch. This is maybe not worth the trouble. + + * Domain-less CPD entries. This is only applicable to write-back caches +that do not require a TLB access to retrieve the physical address of a +write-back such at the ARM9 with it's TAG-RAM. What we can do is tag the CPD +entry with a inactive domain (reserved for this task) effectively disabling the +region and allowing the entries to get cleaned by normal opperation. These +domain-list CPD entries can then be cleared removed on a cache flush. Very +experimental idea, needs a lot more thought and probably won't work. Needs an +ARM9 core too. This actually works on the StrongARM as well. I need to think +about it more though. + + * Anything I've forgotton, basically tuning the implementation and +experimenting with idea's for choosing a domain to preempt for recycling, +choosing an ARM PID, and choosing 1MB regions for mmap(). diff -ruN linux-2.4.24-uc0-snap1/Makefile linux-2.4.24-uc0-snap1-fass0/Makefile --- linux-2.4.24-uc0-snap1/Makefile 2004-02-17 23:52:16.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/Makefile 2004-03-29 09:03:53.000000000 +1000 @@ -1,7 +1,7 @@ VERSION = 2 PATCHLEVEL = 4 SUBLEVEL = 24 -EXTRAVERSION = -uc0 +EXTRAVERSION = -uc0-fass0 KERNELRELEASE=$(VERSION).$(PATCHLEVEL).$(SUBLEVEL)$(EXTRAVERSION) diff -ruN linux-2.4.24-uc0-snap1/arch/arm/kernel/entry-armv.S linux-2.4.24-uc0-snap1-fass0/arch/arm/kernel/entry-armv.S --- linux-2.4.24-uc0-snap1/arch/arm/kernel/entry-armv.S 2004-02-17 23:52:17.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/kernel/entry-armv.S 2004-03-29 09:02:55.000000000 +1000 @@ -1278,13 +1278,20 @@ #endif str sp, [r0, #TSS_SAVE] @ Save sp_SVC ldr sp, [r1, #TSS_SAVE] @ Get saved sp_SVC - ldr r2, [r1, #TSS_DOMAIN] + ldr r3, [r1, #TSK_MM] @ Get tasks mm_struct* + ldr r2, [r1, #TSS_DACR] @ Get task DACR + teq r3, #0 @ Have mm_struct? + ldrne r1, [r3, #MM_CONTEXT_PID] @ Get mm contexts PID + ldrne r3, [r3, #MM_CONTEXT_DACR] @ Get mm contexts DACR + moveq r1, #0 @ Set PID to zero + orr r2, r2, r3 @ Build CPU DACR + mcr p15, 0, r1, c13, c0 @ Set CPU PID #ifdef CONFIG_CPU_XSCALE ldmfd sp!, {r4, r5} mar acc0, r4, r5 #endif ldr ip, [sp], #4 - mcr p15, 0, r2, c3, c0 @ Set domain register + mcr p15, 0, r2, c3, c0 @ Set CPU DACR msr spsr, ip @ Save tasks CPSR into SPSR for this return ldmfd sp!, {r4 - sl, fp, pc}^ @ Load all regs saved previously diff -ruN linux-2.4.24-uc0-snap1/arch/arm/mm/Makefile linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/Makefile --- linux-2.4.24-uc0-snap1/arch/arm/mm/Makefile 2004-02-17 23:52:18.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/Makefile 2004-03-29 09:02:55.000000000 +1000 @@ -20,7 +20,7 @@ export-objs := proc-syms.o discontig.o ifeq ($(CONFIG_CPU_32),y) -obj-y += consistent.o fault-armv.o ioremap.o mm-armv.o +obj-y += consistent.o fault-armv.o ioremap.o mm-armv.o cpd.o pid.o obj-$(CONFIG_MODULES) += proc-syms.o endif diff -ruN linux-2.4.24-uc0-snap1/arch/arm/mm/cpd.c linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/cpd.c --- linux-2.4.24-uc0-snap1/arch/arm/mm/cpd.c 1970-01-01 10:00:00.000000000 +1000 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/cpd.c 2004-03-29 09:02:55.000000000 +1000 @@ -0,0 +1,1096 @@ +/******************************************************************************* +* Filename: arch/arm/mm/cpd.c * +* Description: Caching Page Directory (CPD) functions for Fast Address Space * +* Switching (FASS) in ARM Linux. * +* Created: 20/06/2001 * +* Changes: 14/10/2001 - Added code to deal with ARM PID reloation in * +* cpd_load() and the cpd_*_flush_(page/range)() code. * +* 19/02/2002 - Modified arch_get_unmapped_area() to be more * +* intelligent and added support functions to bring this about. * +* 22/02/2002 - Modifed domain manipulation to add cpd_stats house * +* keeping information. cpd_unallocate_domain() moved to list * +* traversal of CPD. * +* 19/03/2002 - Start of modifications to support shared domains. * +* 25/04/2002 - Changed cpd_allocate_domain() to simply use a * +* round-robin algorithm on the third pass, ie no clean domains. * +* This improved performance during domain thrashing to better then* +* normal linux. * +* 26/04/2002 - Change per address-space domain disabling, via the * +* DACR, to be lazy rather then diabling all domains on a context * +* switch. Now all domains only diabled if a cache/tlb clean event * +* occured since the address-space last ran or if a domain was * +* unallocated. * +* 28/04/2002 - Replaced round-robin domain recycling algorithm * +* with a clock algorithm. * +* Copyright: (C) 2001, 2002 Adam Wiggins * +******************************************************************************** +* Notes: * +* - Cache/TLB flushes don't do anything if no domain is allocated since no * +* user memory can be touched without a domain, similarly if the allocated * +* domain is marked as clean. * +******************************************************************************** +* This program is free software; you can redistribute it and/or modify * +* it under the terms of the GNU General Public Licence version 2 as * +* published by the Free Software Foundation. * +*******************************************************************************/ + +#include +#include +#include +#include +#include +#include + +/********** +* Globals * +**********/ + +/* Used to lazily update address-space DACRs */ +static unsigned long long domain_clock; + +/* The next domain to be allocated in when domain thrashing */ +static int next_domain = DOMAIN_START; + +/* The next 1MB region for mmap()s. Hack for now */ +static int next_unmapped = PAGE_ALIGN(CPD_UNMAPPED_BASE); + +/* Domain TLB/Cache dirty bits, Cleared when coherent */ +unsigned long domain_tlb_coherence = DIRTY; +unsigned long domain_dcache_coherence = DIRTY; +unsigned long domain_icache_coherence = DIRTY; + +unsigned long domain_touched = DIRTY; /* Used by domain recycling clock algo */ + +/* Array of domain_structs for ARM's 16 domains */ +static struct domain_struct domains[DOMAIN_MAX] = { + {number:0}, + {number:1}, + {number:2}, + {number:3}, + {number:4}, + {number:5}, + {number:6}, + {number:7}, + {number:8}, + {number:9}, + {number:10}, + {number:11}, + {number:12}, + {number:13}, + {number:14}, + {number:15} +}; + +/* Statistics on CPD entries */ +struct cpd_struct cpd_stats[USER_PTRS_PER_PGD]; + +/************ +* Functions * +************/ + +/* cpd_switch_mm: This function switches address-space contexts. + * + * Notes: + * - At the moment all domains are disabled on a context switch and reenabled + * via the domain faults. This might need to be changed as the cost is + * noticable, if small. The problem with simply setting it to enable only + * dirty domains means we don't test if a domain has been deallocated, + * reallocated and touched. It either needs to be tested here or revoked + * from all address spaces at deallocation time. Need to think about it a + * little more. + */ +void +cpd_switch_mm(struct mm_struct* mm_p) +{ + +#if LAZY_DISABLE + + if(mm_p->context.domain_timestamp != domain_clock) { + + //printk("** cpd_switch: tlb 0x%x icache 0x%x dcache 0x%x dacr 0x%x**\n", + // domain_tlb_coherence, domain_icache_coherence, domain_dcache_coherence, + // get_dacr()); + + + /* Disable 'mm_p's domains that are clean or recycled */ + mm_p->context.dacr = 0; /* disables all domains */ + mm_p->context.domain_timestamp = domain_clock; + } + +#else + + /* Disable 'mm_p's domains that are clean or recycled */ + mm_p->context.dacr = 0; /* disables all domains */ + +#endif + +#if CLOCK_RECYCLE + + domain_touched |= mm_p->context.dacr; + +#endif + +} /* cpd_switch_mm() */ + +/* cpd_load: This function loads a Page Directory Entry 'pmd' from the 'current' + * process into the CPD. It simply takes 'pmd' and tags it with the domain + * allocated to the region it is associated with. The result is then loaded + * into the required CPD entry. The function updates the cpd_count of the + * domain and does a coherence check if the CPD entry is being replaced, ie + * it was previously allocated to another region. + * Notes: + * - mv_addr is the ARM PID relocated Modifed Virtual Address (MVA) seen by + * the FASS part of the kernel. Since the CPD is the only pg_dir walked by + * the hardware it is the only pg_dir that needs to do ARM PID relocation. + */ +void +cpd_load(pmd_t pmd, unsigned long mv_addr) +{ + struct domain_struct* domain_p = + cpd_get_domain(current->mm, mva_to_va(mv_addr, current->mm)); + pmd_t* cpd_p = pmd_offset(pgd_offset_k(mv_addr), mv_addr); + unsigned int index = CPD_P_TO_INDEX(cpd_p); + int old_domain = pmd_domain(*cpd_p); + + //printk("** cpd_load: i %d old %d new %d **\n", index, old_domain, + // domain_p->number); + + if(old_domain) { /* CPD entry being replaced, TLB/Cache coherence needed */ + + assert(old_domain != domain_p->number); + + /* update cpd_stats[] */ + + cpd_stats[index].collisions++; + cpd_stats_del(old_domain, cpd_stats + index); + + /* update domains[] */ + + domains[old_domain].cpd_count--; + + if(!cpd_is_domain_cache_coherent(old_domain)) { + + cpd_cache_clean(); + } /* The cache clean is not required for ARM9, see cpd.h Notes */ + + //if(!cpd_is_domain_tlb_coherent(old_domain)) { + // This should be made into a targeted TLB flush. + + cpd_tlb_clean(); + //} + } + + /* Update domains[] */ + + domain_p->cpd_count++; + cpd_stats_add(domain_p, cpd_stats + index); + + /* Update CPD */ + update: + + pmd_val(pmd) |= PMD_DOMAIN(domain_p->number); /* Tag CPD entry */ + cpu_set_pmd(cpd_p, pmd); + +} /* cpd_load() */ + +/* cpd_allocate_region: ? + */ +struct region_struct* +cpd_allocate_region(void) +{ + struct region_struct* region_p = kmalloc(sizeof(struct region_struct), + GFP_KERNEL); /* Is this correct? */ + + if(region_p) { + region_p->mm_count = 1; + region_p->domain_p = NULL; + region_p->next = NULL; + } + + return region_p; + +} /* cpd_allocate_region() */ + +/* cpd_unallocate_region: Free's a region. + * + * Notes: + * - Using the non-lazy cache/tlb flushing because the lazy versions crash + * the kernel, have to come back and have a look at this later. + */ +void +cpd_unallocate_region(struct region_struct* region_p) +{ + struct domain_struct* domain_p = region_p->domain_p; + + /* Free domain if we have one */ + if(domain_p) { + + /* Make domain coherent */ + if(!cpd_is_domain_cache_coherent(domain_p->number)) { + cpu_cache_clean_invalidate_all(); /* avoiding lazy flush, see below */ + } + + if(!cpd_is_domain_tlb_coherent(domain_p->number)) { + cpu_tlb_invalidate_all(); /* avoiding lazy flush because it crashes */ + } + + /* Unallocate domain */ + cpd_unallocate_domain(domain_p); + } + + /* Free the structure */ + kfree(region_p); + +} /* cpd_unallocate_region() */ + +/* cpd_get_region: This function retrieves the region allocated to the + * address-space at v_addr. + */ +struct region_struct* +cpd_get_region(struct mm_struct* mm_p, unsigned int v_addr) +{ + struct vm_area_struct* vma_p = find_vma(mm_p, v_addr); + + if(vma_p && (v_addr >= vma_p->vm_start) && vma_p->vm_sharing_data) { + return vma_p->vm_sharing_data; + + } else { + return mm_p->context.region_p; + } + +} /* cpd_get_region() */ + +/* cpd_get_active_domain: This function simply returns the domain number + * allocated to 'mm_p' in the region covered by 'v_addr'. If no domain is + * allocated 0 is returned. 'v_addr' == NULL requests the private domain of + * 'mm_p'. + * + * Notes: + * * Currently just returns the private domain, needs fixing. + */ +int +cpd_get_active_domain(struct mm_struct* mm_p, unsigned int v_addr) +{ + struct region_struct* region_p; + + if(mm_p) { + region_p = cpd_get_region(mm_p, v_addr); + + return region_p->domain_p ? region_p->domain_p->number : 0; + + } else { + + return 0; + } + +} /* cpd_get_active_domain() */ + +/* cpd_get_domain: This function retrieves the domain currently allocated to the + * given region. If there is none, then one is allocated, possibly preempted + * from another region. + * + * Notes: + * * Currently just returns the private domain, needs fixing. + */ +struct domain_struct* +cpd_get_domain(struct mm_struct* mm_p, unsigned int v_addr) +{ + struct region_struct* region_p = cpd_get_region(mm_p, v_addr); + + return region_p->domain_p ? region_p->domain_p : + cpd_allocate_domain(region_p); + +} /* cpd_get_domain() */ + +/* cpd_allocate_domain: This function finds a free domain or preempts a used + * one. A victim domain is selected with the following criteria: + * - The first unallocated domain is used. OR + * - The domain with the least CPD entries, which doesn't require a cache/TLB + * flush. OR + * - The domain with the least CPD entries. + * Notes: + * - A policy of replacing the domain with the least CPD entries might not be + * the best. A round robbin or other policy may be more appropriate. This + * needs further thought and investigation. + */ +struct domain_struct* +cpd_allocate_domain(struct region_struct* region_p) +{ + int i; + int new_domain; + struct domain_struct* domain_p; + + /* First pass to find an unallocated domain */ + for(i = DOMAIN_START; i <= DOMAIN_END; i++) { + if(!domains[i].region_p) { + domain_p = domains + i; + goto allocate; + } + } + +#if CLOCK_RECYCLE + + /* First run of the clock domain_usage might not equal zero */ + clock: + + /* Second pass to find a clean domain with the least entries in the CPD */ + for(i = DOMAIN_START; i <= DOMAIN_END; i++) { + new_domain = next_domain; + (next_domain == DOMAIN_END) ? (next_domain = DOMAIN_START) : next_domain++; + + if(cpd_is_domain_coherent(next_domain) && + !cpd_domain_touched(next_domain)) { + domain_p = domains + next_domain; + + goto unallocate; + } + } + + /* Third pass to find a domain using clock scheme */ + for(i = DOMAIN_START; i <= DOMAIN_END; i++) { + new_domain = next_domain; + (next_domain == DOMAIN_END) ? (next_domain = DOMAIN_START) : next_domain++; + + if(!cpd_domain_touched(next_domain)) { + domain_p = domains + next_domain; + + goto flush; + } + + cpd_clear_domain_touched(next_domain); + } + + /* Now, domain_usage == 0 */ + goto clock; + + /* Unallocated domain needs to be made coherent */ + flush: + + /* The old third pass leads to very poor behaviour in the case that domains + are thrashing, ie about 3-4 times the number of domains in processes are + heavily thrashing. + */ + /* Third pass to find a domain with the least entries in the CPD */ + //for(i = new_domain = DOMAIN_START; i <= DOMAIN_END; i++) { + // if(domains[i].cpd_count < domains[new_domain].cpd_count) { + // new_domain = i; + // } + //} + +#else /* Round-robin recycling algorithm */ + + /* Second pass to find a clean domain with the least entries in the CPD */ + for(i = new_domain = DOMAIN_START; i <= DOMAIN_END; i++) { + if(cpd_is_domain_coherent(i) && + ((domains[i].cpd_count * domains[i].region_p->mm_count) < + (domains[new_domain].cpd_count * domains[i].region_p->mm_count))) { + + domain_p = domains + i; + + goto unallocate; + } + } + + /* Third pass to find a domain in round-robin scheme when thrashing. */ + + new_domain = next_domain; + (next_domain == DOMAIN_END) ? next_domain = DOMAIN_START : next_domain++; + +#endif + + domain_p = domains + new_domain; + + /* Domain has entries in CPD, clean caches and TLB */ + if(!cpd_is_domain_cache_coherent(new_domain)) { + cpd_cache_clean(); + } + + if(!cpd_is_domain_tlb_coherent(new_domain)) { + cpd_tlb_clean(); + } + + unallocate: + cpd_unallocate_domain(domain_p); + + allocate: + + domain_p->region_p = region_p; + region_p->domain_p = domain_p; + + //printk("** cpd_allocate_domain: %d **\n", domain_p->number); + + return domain_p; + +} /* cpd_allocate_domain() */ + +/* cpd_unallocate_domain: Insures all CPD entries owned by 'domain_p' are + * removed from CPD and the domain is unallocated. + * Assumptions: + * - domain_p != NULL. + * - domain_p->region_p != NULL; + * - TLBs coherent, ie no TLB entries tagged with this domain. + * Notes: + * - Should domain and region pointer updates be atomic? + */ +void +cpd_unallocate_domain(struct domain_struct* domain_p) +{ + int i; + pmd_t* cpd_p = pmd_offset(pgd_offset_k(0), 0); /* Get CPD Address */ + struct cpd_struct* cpd_stat_p; + + //printk("** cpd_unallocate_domain: %d **\n", domain_p->number); + + /* Loop through CPD entries used by this domain */ + for(cpd_stat_p = domain_p->cpd_head; cpd_stat_p != NULL; + cpd_stat_p = cpd_stat_p->cpd_next) { + + i = STATS_P_TO_INDEX(cpd_stat_p); + + /* Update CPD */ + cpu_set_pmd(cpd_p + i, __pmd(0)); + + /* Update domain_p */ + domain_p->cpd_count--; + } + + /* Remove unallocate domain from address-space context */ + + domain_p->cpd_head = NULL; + domain_p->region_p->domain_p = NULL; + domain_p->region_p = NULL; + +#if LAZY_DISABLE + + /* Update the domain clock since we unallocated a domain */ + domain_clock++; + +#endif + + //dump_cpd(1); + + set_dacr(current->thread.dacr); + +} /* cpd_unallocate_domain() */ + +void +arch_remove_shared_vm_struct(struct vm_area_struct* vma_p) +{ + struct region_struct* region_p; + + /* If we have a region assigned update the count */ + if((region_p = (struct region_struct*)vma_p->vm_sharing_data) != NULL) { + region_p->mm_count--; + + } else { + + return; + } + + /* Free the shared region if this is the last user */ + if(!region_p->mm_count) { + cpd_unallocate_region(region_p); + } + +} /* arch_remove_shared_vm_struct() */ + +/* arch_get_unmapped_area: To minimise conflicts in the CPD, hence minimise + * cache/TLB flushes. Mappings are split into two categories and the relevant + * handler is called to allocate an address: + * - Shared mappings. + * - Private mappings. + */ +inline unsigned long +arch_get_unmapped_area(struct file *file_p, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags) +{ + if(len > (TASK_SIZE - CPD_UNMAPPED_BASE)) + return -ENOMEM; + + if(flags & MAP_SHARED) { + return cpd_get_shared_area(file_p, addr, len, pgoff, flags); + } + else { + return cpd_get_private_area(file_p, addr, len, pgoff, flags); + } +} /* arch_get_unmapped_area() */ + +/* cpd_next_area: Allocates a starting point to search from. + * Notes: + * - Currently ignores the addr argument, might remove. + */ +inline unsigned long +cpd_next_area(unsigned long given_addr) +{ + unsigned long addr = next_unmapped; + + next_unmapped = (next_unmapped >= (TASK_SIZE - MEGABYTE(1)) ? + PAGE_ALIGN(CPD_UNMAPPED_BASE) : next_unmapped + MEGABYTE(1)); + + return addr; +} + +/* cpd_get_shared_area: This function tries to allocate the shared mapping to + * the same address it is allocated to in other address spaces. If it cannot + * or this is the first instance of the shared mapping it will allocate a new + * unique area. + * Notes: + * - Not sure if we need to check file_p. Can we have shared mappings without + * a file linking them? + * - If shared domains are used this will promote the situation where the + * shared mappings are at the same memory location which is what we want to + * avoid aliases. + * - This will lead to incorrect behaviour as really to be truely sharing + * the length of the new and existing mappings have to be the same and the + * perms have to be the same otherwise one of the address-spaces gets to see + * more or get to do more with it. This is limited by max len/permissions of + * the set of address-spaces sharing the mapping at this VA. + */ +unsigned long +cpd_get_shared_area(struct file *file_p, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + struct vm_area_struct* vma_priv_p; /* VMA in current address-space */ + struct vm_area_struct* vma_shrd_p; /* VMA of address-space sharing mapping */ + + if(file_p) { + + for(vma_shrd_p = file_p->f_dentry->d_inode->i_mapping->i_mmap_shared; + vma_shrd_p; vma_shrd_p = vma_shrd_p->vm_next_share) { + + /* Check we are looking at the 'same' part of the file */ + if(vma_shrd_p->vm_pgoff == pgoff) { + + addr = vma_shrd_p->vm_start; + vma_priv_p = find_vma(current->mm, addr); + + /* Will it fit in current address space? */ + if(!vma_priv_p || ((addr + len) <= vma_priv_p->vm_start)) { + + /* It does so use the same virtual address */ + return addr; + } + } + } + + /* This is the first mapping for the file so allocate a unique area */ + return cpd_get_unique_area(file_p, addr, len, pgoff, flags); + } + + /* Can't share so simply map private */ + return cpd_get_private_area(file_p, addr, len, pgoff, flags); + +} /* cpd_get_shared_area() */ + +/* cpd_get_unique_area: This function allocates a unique 1MB region for the + * in the processes address space. It tries to allocate a region which will + * not lead to CPD conflicts. + * Notes: + * - Currently it simply cycles through 1MB regions between CPD_UNMAPPED_BASE + * and TASK_SIZE. This should be made smarter to take into account the + * following: + * * The contention of the 1MB region. A 1MB region already allocated to + * another task is probably a bad idea since it will lead to CPD + * conflicts. + */ +unsigned long +cpd_get_unique_area(struct file *file_p, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + char wrap = 0; + char dense = 0; + unsigned long start = cpd_next_area(addr); + struct vm_area_struct *vma_p; + + addr = start; + + for(vma_p = find_vma(current->mm, addr);; + addr = vma_p->vm_end, vma_p = vma_p->vm_next) { + + /* At this point: (!vma || addr < vma->vm_end). */ + if((TASK_SIZE - len) < addr) { /* Wrap around */ + + addr = PAGE_ALIGN(CPD_UNMAPPED_BASE); + wrap = 1; + } + + if(wrap && ((start - len) < addr)) { + if(dense) { /* Mapping doesn't fit */ + + return -ENOMEM; + } + + wrap = 0; + dense = 1; /* Start 2nd run, buddy up in 1MB slots already being used */ + } + + if(!dense && vma_p && + (MASK_MEGABYTE(addr, 1) == MASK_MEGABYTE(vma_p->vm_start, 1))) { + + continue; /* First run try keep mapping in a unique 1MB slot */ + } + + if(!vma_p || addr + len <= vma_p->vm_start) { + + return addr; + } + } + +} /* cpd_get_unique_area() */ + +/* cpd_get_private_area: This function attempts to allocate an address inside + * the process 32MB relocated area of its address space. If no suitable + * region is found cpd_get_alloced_area() is called. + * + * Notes: + * - vma_p will never be NULL as a 4KB dummy VMA is mapped at 31MB. + * - vma__prev_p will never be NULL as the brk vm_area is there or the loop + * invarient has failed. + */ +unsigned long +cpd_get_private_area(struct file *file_p, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + unsigned int start, end; + struct vm_area_struct* vma_p; + struct vm_area_struct* vma_prev_p; + + if(len < ARMPID_BRK_LIMIT) { + + /* Loop till we hit brk */ + for(start = end = ARMPID_BRK_LIMIT; start >= PAGE_SIZE; + end = vma_prev_p->vm_start) { + + if(len > end) { + break; + } + + start = end - len; + vma_p = find_vma_prev(current->mm, end, &vma_prev_p); + + if(start >= vma_prev_p->vm_end) { + + return start; + } + + if(!vma_prev_p) { + break; + } + } + } + + return cpd_get_alloced_area(file_p, addr, len, pgoff, flags); + +} /* cpd_get_private_alloced_area() */ + +/* cpd_get_alloced_area: This function attempts to allocate a 1MB region already + * in use by the process that the new mapping fits into. If no suitable + * region is found a unique one is allocated. + */ +unsigned long +cpd_get_alloced_area(struct file *file_p, unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + unsigned long slot; /* 1MB slot being checked */ + struct vm_area_struct* vma_p; + + if(len <= MEGABYTE(1)) { /* Allocations of 1MB or more get a unique area */ + + addr = CPD_UNMAPPED_BASE; + + /* For each vm_area see if we can buddy up */ + for(vma_p = find_vma(current->mm, addr); vma_p; + addr = vma_p->vm_end, vma_p = vma_p->vm_next) { + + slot = MASK_MEGABYTE(vma_p->vm_start, 1); /* vm_area's 1MB start slot */ + if(addr < slot) { /* Seek to 1MB's slots start if not already in it */ + addr = slot; + } + + /* Try to allocate in the 1MB the vm_area starts in */ + if((addr + len) <= vma_p->vm_start) { /* Fits */ + + return addr; + } + } + } + + return cpd_get_unique_area(file_p, addr, len, pgoff, flags); + +} /* cpd_get_alloced_area() */ + +/* cpd_set_pmd: Insures coherence of CPD with user pmd modifications. + * Assumptions: + * - CPD contains kernel mappings which are always coherent. + * - User changes are coherent if the old entry is not in the CPD. + * - The pmd being set belongs to 'current->mm'. I haven't verified this + * though. Please let me know if this assumption is (in)correct. + * Action: + * - If the CPD entry coresponding to 'pmd_p' is tagged with the same domain + * as 'pmd_p's region then update it. If current->mm doesn't exist the CPD + * is uneffected. + * Notes: + * - This function is a bit of a mess, could do with a clean up. + * - The TLBs/Caches are flushed here because the CPD entry is zero'd. + * - Not totally sure the cache flush is needed as an unmapping would probably + * have already flushed the cache. However that marks it clean so this flush + * would be avoided... + * - zero'ing the CPD entry could be replaced with actually updating it. Would + * require an extra check. However this function is only going to be used + * when a new pmd is (un)allocated, in one case it couldn't have been in the + * CPD and in the other it is removed from the CPD anyway. + */ +void +cpd_set_pmd(pmd_t* pmd_p, pmd_t pmd) +{ + pmd_t* cpd_p; + int domain_cpd, domain_pmd; + struct mm_struct* mm_p = current->mm; + unsigned long index = CPD_P_TO_INDEX(pmd_p); + + if(index > USER_PTRS_PER_PGD) { /* If pmd is kernel space mapping */ + cpu_set_pmd(pmd_p, pmd); + } + else { /* Must make CPD coherent */ + + if(index < ARMPID_PTRS_PER_PGD && mm_p) {/* Adjust for ARM PID relocation */ + index = index + ARMPID_CPD_OFFSET(mm_p->context.pid); + } + + cpd_p = pmd_offset(pgd_offset_k(0), 0) + index; + domain_cpd = pmd_domain(*cpd_p); + domain_pmd = cpd_get_active_domain(mm_p, MEGABYTE(index)); + + if(domain_cpd && (domain_cpd == domain_pmd)) { + + /* update cpd_stats[] */ + + cpd_stats[index].collisions++; /* Not really a collision */ + cpd_stats_del(domain_cpd, cpd_stats + index); + + domains[domain_cpd].cpd_count--; + + if(!cpd_is_domain_cache_coherent(domain_cpd)) { + cpd_cache_clean(); + } + + cpu_set_pmd(cpd_p, __pmd(0)); + + if(!cpd_is_domain_tlb_coherent(domain_cpd)) { + cpd_tlb_clean(); + } + } + + *pmd_p = pmd; /* Hardware never accesses user pg_dirs, only cpd code does */ + } + +} /* cpd_set_pmd() */ + +/* cpd_tlb_flush_all: Insures coherence of all TLB entries owned by the kernel, + * ie addresses above PAGE_OFFSET. + * Assumptions: + * - Only used to make kernel mappings coherent. + * - Kerenl mappings in CPD are always coherent since it is the master pg_dir. + * - Kernel mappings are not coherent with respect to the TLB. + * Action: + * - Flush all TLB entries. + */ +void +cpd_tlb_flush_all(void) +{ + cpd_tlb_clean(); + +} /* cpd_tlb_flush_all() */ + +/* cpd_tlb_flush_mm: Insures coherence of all TLB entries entries owned by + * 'mm_p' with its page table. + * Assumptions: + * - No kernel mapping changed so TLB coherent with respect to them. + * - User TLB entries covered by CPD entries owned by 'mm_p' are not coherent. + * - CPD entries are updated if required via set_pmd(). + * Action: + * - If any CPD entries are tagged with 'mm_p's domain, if any, flush all TLB + * entries. + */ +void +cpd_tlb_flush_mm(struct mm_struct* mm_p) +{ + if(!cpd_is_mm_tlb_coherent(mm_p)) { + cpd_tlb_clean(); + } +} /* cpd_tlb_flush_mm() */ + +/* cpd_tlb_flush_range: Insures coherence of all TLB entries owned by 'mm_p' in + * the range 'va_start' to 'va_end'. + * Assumptions: + * - No kernel mappings effected so TLB coherent with respect to them. + * - CPD coherent with task pg_dir entries, via set_pmd(). + * - TLB entries covered by CPD entries owned by 'mm_p' are not coherent over + * the range. + * Action: + * - If any entries in the CPD covered by the range are tagged with 'mm_p's + * domain, clean TLBs. + * Notes: + * - cpd_tlb_clean() is used instead of cpu_tlb_invalidate_range() since it + * can mark the TLBs as clean. This should maybe be changed to only do + * cpd_tlb_clean() if the range is a significant portion of the TLB + * coverage. + * - The range is specified in VA's while the flushing call and CPD accesses + * use MVA's. + */ +void +cpd_tlb_flush_range(struct mm_struct* mm_p, + unsigned long va_start, unsigned long va_end) +{ + int domain; + unsigned long mva_start = va_to_mva(va_start, mm_p); + unsigned long mva_end = va_to_mva(va_end, mm_p); + pmd_t* cpd_end_p = pmd_offset(pgd_offset_k(mva_end), mva_end); + pmd_t* cpd_start_p = pmd_offset(pgd_offset_k(mva_start), mva_start); + + if(!cpd_is_mm_tlb_coherent(mm_p)) { /* 'mm_p' has dirty CPD entries */ + while(cpd_start_p <= cpd_end_p) { + + domain = pmd_domain(*cpd_start_p); + + if(!cpd_is_domain_tlb_coherent(domain) && + domain_active(mm_p->context.dacr, domain)) { + + cpd_tlb_clean(); + + return; /* Flush TLB only once */ + } + + cpd_start_p++; + } + } +} /* cpd_tlb_flush_range() */ + +/* cpd_tlb_flush_page: Insures coherence of TLB entry owned by 'vma_p's mm for + * 'va_page'. + * Assumptions: + * - Mapping is not a kernel one so TLB is coherent with respect to kernel + * mappins. + * - User CPD entry coherent, via set_pmd() and only a single page effected. + * - TLB entry mapping 'va_page' is not coherent iff covered by a CPD entry + * owned by the mm associated with 'vma_p'. + * Action: + * - if CPD covering 'va_page' is owned by 'vma_p's mm, invalidate TLB + * entries. + * Notes: + * - The page is specified by a VA while the flushing call and CPD access + * uses a MVA. + */ +void +cpd_tlb_flush_page(struct vm_area_struct* vma_p, unsigned long va_page) +{ + pmd_t cpd; + int domain; + unsigned long mva_page = va_to_mva(va_page, vma_p->vm_mm); + + /* Does 'vma_p's mm have any incoherencies? */ + if(~cpd_is_mm_tlb_coherent(vma_p->vm_mm)) { + cpd = *pmd_offset(pgd_offset_k(mva_page), mva_page); + domain = pmd_domain(cpd); + + /* Is CPD entry's domain incoherent and active in 'vma_p's mm? */ + if(!cpd_is_domain_tlb_coherent(domain) && + domain_active(vma_p->vm_mm->context.dacr, domain)) { + + cpu_tlb_invalidate_page(mva_page, vma_p->vm_flags & VM_EXEC); + } + } +} /* cpd_tlb_flush_page() */ + +/* cpd_cache_flush_all: Insures coherence of all cache entries owned by the + * kernel, ie addresses above PAGE_OFFSET. + * Assumptions: + * - Only use to make kernel memory coherent. + * Action: + * - Clean/Flush all caches. + */ +void +cpd_cache_flush_all(void) +{ + cpd_cache_clean(); + +} /* cpd_cache_flush_all() */ + +/* cpd_cache_flush_mm: Insures coherence of all cache entries owned by 'mm_p'. + * Assumptions: + * - Kernel memory coherent with caches. + * - User memory covered by CPD entries owned by 'mm_p' are not coherent with + * caches. + * Action; + * - If any CPD entries are tagged with 'mm_p's domain, if any, clean/flush + * all cache entries. + */ +void +cpd_cache_flush_mm(struct mm_struct* mm_p) +{ + if(!cpd_is_mm_cache_coherent(mm_p)) { + cpd_cache_clean(); + } +} /* cpd_cache_flush_mm() */ + +/* cpd_cache_flush_range: Insures coherence of all cache entries owned by 'mm_p' + * in the range 'va_start' to 'va_end'. + * Assumptions: + * - Kernel memory coherent with caches. + * - Cache entries covered by CPD entries owned by 'mm_p' are not coherent + * over the range. + * Action: + * - If any entries in the CPD covered by the range are tagged with 'mm_p's + * domain, clean/flush all cache entries. + * Notes: + * - cpd_cache_clean() is used instead of cpu_cache_clean_invalidate_range() + * since it can mark the caches as clean. This should maybe be changed to + * only do cpd_cache_clean() if the range is a significant portion of the + * cache. + * - The range is specified in VA's while the flushing call and CPD accesses + * use MVA's. + */ +void +cpd_cache_flush_range(struct mm_struct* mm_p, + unsigned long va_start, unsigned long va_end) +{ + int domain; + unsigned long mva_start = va_to_mva(va_start, mm_p); + unsigned long mva_end = va_to_mva(va_end, mm_p); + pmd_t* cpd_end_p = pmd_offset(pgd_offset_k(mva_end), mva_end); + pmd_t* cpd_start_p = pmd_offset(pgd_offset_k(mva_start), mva_start); + + if(!cpd_is_mm_cache_coherent(mm_p)) { /* 'mm_p' has dirty CPD entries */ + while(cpd_start_p <= cpd_end_p) { + + domain = pmd_domain(*cpd_start_p); + + if(!cpd_is_domain_cache_coherent(domain) && + domain_active(mm_p->context.dacr, domain)) { + + cpd_cache_clean(); + + return; /* Flush caches only once */ + } + + cpd_start_p++; + } + } +} /* cpd_cache_flush_range() */ + +/* cpd_cache_flush_page: Insures coherent of cache entries owned by 'vma_p's mm + * for 'va_page'. + * Assumptions: + * - Kernel memory coherent with caches. + * - User caches entries covered by 'va_page' are not coherent iff covered by + * a CPD entry owned by the mm associated with 'vma_p'. + * Action: + * - if CPD covering 'va_page' is owned by 'vma_p's mm, invalidate caches + * entries. + * Notes: + * - The page is specified by a VA while the flushing call and CPD access + * uses a MVA. + */ +void +cpd_cache_flush_page(struct vm_area_struct* vma_p, unsigned long va_page) +{ + pmd_t cpd; + int domain; + unsigned long mva_page = va_to_mva(va_page, vma_p->vm_mm); + + /* Does 'vma_p's mm have any incoherencies? */ + if(~cpd_is_mm_cache_coherent(vma_p->vm_mm)) { + cpd = *pmd_offset(pgd_offset_k(mva_page), mva_page); + domain = pmd_domain(cpd); + + /* Is CPD entry's domain incoherent and active in 'vma_p's mm? */ + if(!cpd_is_domain_cache_coherent(domain) && + domain_active(vma_p->vm_mm->context.dacr, domain)) { + + cpu_cache_clean_invalidate_range(mva_page, mva_page + PAGE_SIZE, + vma_p->vm_flags & VM_EXEC); + } + } + +} /* cpd_cache_flush_page() */ + +/* dump_cpd: Dumps the entire CPD for debugging. + * Notes: + * - user = 0, all entries dumped; user = 1 only user entries dumped. + user = -1, dump domain 0 entries only + */ +void +dump_cpd(int user) +{ + int i; + int domain; + pmd_t *cpd_p = pmd_offset(pgd_offset_k(0), 0); /* Get CPD Address */ + + for(i = 0; i < PTRS_PER_PGD; i++) { + domain = pmd_domain(cpd_p[i]); + if((user == -1) && (domain = 0) && pmd_val(cpd_p[i])) { + printk("** dump_cpd() cpd[%d] 0x%x domain 0 **\n", + i, (unsigned int)pmd_val(cpd_p[i])); + continue; + } + if(!user || (domain >= DOMAIN_START && domain <= DOMAIN_END)) { + printk("** dump_cpd() cpd[%d] 0x%x domain tag %d **\n", + i, (unsigned int)pmd_val(cpd_p[i]), domain); + } + } + printk("\n"); +} /* dump_cpd() */ + +/* dump_cpd_stat: Dumps info from a cpd_struct */ +void +dump_cpd_stat(struct cpd_struct* cpd_stat_p) +{ + printk("** dump_cpd_stat: cpd_stat_p 0x%x collisions %d **\n", + (unsigned int)cpd_stat_p, (unsigned int)cpd_stat_p->collisions); + printk("** dump_cpd_stat: cpd_next 0x%x cpd_prev 0x%x **\n", + (unsigned int)cpd_stat_p->cpd_next, + (unsigned int)cpd_stat_p->cpd_prev); + +} /* dump_cpd_stat() */ + +/* dump_cpd_stats: Dump all cpd_stats associated with a domain */ +void +dump_cpd_stats(int domain) +{ + struct cpd_struct* cpd_stat_p = domains[domain].cpd_head; + + printk("** dump_cpd_stats: domain %d **\n", domain); + + while(cpd_stat_p != NULL) { + dump_cpd_stat(cpd_stat_p); + + cpd_stat_p = cpd_stat_p->cpd_next; + } +} /* dump_cpd_stats() */ + +/* dump_vma_single: Dumps info for a vm_area struct. */ +void +dump_vma_single(struct vm_area_struct* vma_p) +{ + assert(vma_p); + + printk("** dump_vma_single: start 0x%x end 0x%x **\n", + (unsigned int)vma_p->vm_start, (unsigned int)vma_p->vm_end); + +} /* dump_vma_single() */ + +/* dump_vma: Dump all vma's in a address-space */ +void +dump_vma(struct mm_struct* mm_p) +{ + struct vm_area_struct* vma_p; + + printk("** dump_vma: mm_p 0x%x **\n", (unsigned int)mm_p); + assert(mm_p); + + for(vma_p = mm_p->mmap; vma_p; vma_p = vma_p->vm_next) { + dump_vma_single(vma_p); + } + +} /* dump_vma() */ diff -ruN linux-2.4.24-uc0-snap1/arch/arm/mm/fault-armv.c linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/fault-armv.c --- linux-2.4.24-uc0-snap1/arch/arm/mm/fault-armv.c 2004-02-17 23:52:18.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/fault-armv.c 2004-03-29 13:03:28.000000000 +1000 @@ -3,6 +3,13 @@ * * Copyright (C) 1995 Linus Torvalds * Modifications for ARM processor (c) 1995-2003 Russell King + * Modifications for FASS (C) 2001-2003 Adam Wiggins and John Zaitseff + * + * FASS Notes: + * - The Modified Virtual Address (MVA) is changed into a Virtual Address (VA + * with ARM PID relocation removed) before being forwarded to the + * do_*_fault() handlers. Only the FASS handlers do_(cpd/domain)_fault() + * are aware of the ARM PID relocaiton. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -18,6 +25,8 @@ #include #include +#include +#include #include "fault.h" @@ -42,6 +51,106 @@ return 1; } +/* awiggins + * do_cpd_fault: Handle Caching Page Directory 'invalid' entry miss (ie CPD + * entry is invalid) or a conflict miss (ie another process is using the CPD + * entry). + * Reason: + * (1) CPD entry is just empty, ie not allocate to any process. + * (2) 'current' processes mm context has no section entry (ie no mapping). + * (3) CPD entry is tagged with another processes domain. + * Action: + * (1) Load the section entry from the pg_dir of 'current' process. + * (2) Call do_translation_fault() and add entry if any mapping is granted, + * else the kernel will raise a seg fault. + * (3) Same as (1)/(2) depending on if 'current' process has a mapping in it's + * pg_dir or not. + */ +static int +do_cpd_fault(unsigned long v_addr, unsigned int fsr, struct pt_regs *regs) +{ + int fault; + unsigned long mv_addr = va_to_mva(v_addr, current->mm); + pmd_t pmd = *pmd_offset(pgd_offset(current->mm, v_addr), v_addr); + + //printk("** do_cpd_fault: **\n"); + + /* if there is a valid entry in the pg_dir of 'current' load it into CPD */ + if(!pmd_none(pmd)) { + + goto success; + } + /* Else take the page fault. and try again */ + else if(!(fault = do_translation_fault(v_addr, fsr, regs))) { + pmd = *pmd_offset(pgd_offset(current->mm, v_addr), v_addr); + + goto success; + } + else { + + return fault; + } + + success: + cpd_load(pmd, mv_addr); + + return 0; /* Success */ + +} /* do_cpd_fault() */ + +/* awiggins + * do_domain_fault: Handle domain faults, these are generated by conflict + * misses or threads touching their currently allocated domain for the first + * time. + * Reason: + * (1) CPD entry is tagged with the wrong domain. + * (2) A domain is allocated but it hasn't been activated. + * Action: + * (1) Call do_cpd_fault() to handle the conflict miss. + * (2) Activate allocated domain in the CPU DACR and return. + */ +static int +do_domain_fault(unsigned long v_addr, unsigned int fsr, struct pt_regs *regs) +{ + int domain = cpd_get_active_domain(current->mm, v_addr); + + //printk("** do_domain_fault: **\n"); + + if(domain && !domain_active(get_dacr(), domain)) { + update_dacr(domain); /* modified */ + + return 0; /* Success */ + } + else { + + return do_cpd_fault(v_addr, fsr, regs); + } +} /* do_domain_fault() */ + +/* do_domain_fixup: This is required to fixup page translation faults on a + * shared domain. + */ +static int +do_domain_fixup(unsigned long v_addr, unsigned int fsr, struct pt_regs *regs) +{ + unsigned long mv_addr = va_to_mva(v_addr, current->mm); /* Modified VA */ + pmd_t* cpd_p = pmd_offset(pgd_offset_k(mv_addr), mv_addr); + pmd_t pmd = *pmd_offset(pgd_offset(current->mm, v_addr), v_addr); + + /* This needs to be cleaned up a little */ + if((pmd_val(*cpd_p) >> 10) != (pmd_val(pmd) >> 10)) { + + //printk("** do_domain_fixup: **\n"); + + pmd_val(pmd) |= PMD_DOMAIN(pmd_domain(*cpd_p)); /* Tag CPD entry */ + cpu_set_pmd(cpd_p, pmd); + + } + + return do_page_fault(v_addr, fsr, regs); + +} /* do_domain_fixup() */ + static struct fsr_info { int (*fn)(unsigned long addr, unsigned int fsr, struct pt_regs *regs); int sig; @@ -52,13 +161,13 @@ { do_bad, SIGKILL, "terminal exception" }, { do_bad, SIGILL, "alignment exception" }, { do_bad, SIGBUS, "external abort on linefetch" }, - { do_translation_fault, SIGSEGV, "section translation fault" }, + { do_cpd_fault, SIGSEGV, "section translation fault" }, { do_bad, SIGBUS, "external abort on linefetch" }, - { do_page_fault, SIGSEGV, "page translation fault" }, + { do_domain_fixup, SIGSEGV, "page translation fault" }, { do_bad, SIGBUS, "external abort on non-linefetch" }, - { do_bad, SIGSEGV, "section domain fault" }, + { do_domain_fault, SIGSEGV, "section domain fault" }, { do_bad, SIGBUS, "external abort on non-linefetch" }, - { do_bad, SIGSEGV, "page domain fault" }, + { do_domain_fault, SIGSEGV, "page domain fault" }, { do_bad, SIGBUS, "external abort on translation" }, { do_sect_fault, SIGSEGV, "section permission fault" }, { do_bad, SIGBUS, "external abort on translation" }, @@ -79,23 +188,63 @@ /* * Dispatch a data abort to the relevant handler. */ +/* awiggins +* do_DataAbort: Modified to check for a translation fault on a page that should +* be classed as a domain fault on a page. Tranlsation faults have a higher +* prioirty then domain fault. What can happen is that the CPD entry a process +* touches is not it's own and the 2nd level page table has no entry for the +* faulting address, so you get a translation fault on a page even though your +* not accessing your own pg_dir entry because the domain tag will be ignored. +* In this case we need to flag the fault as a domain fault on a page to +* produce the correct behaviour of replacing the CPD entry to one owned by +* the current process. +* Notes: +* - For data aborts the faulting address is supplied by the ARM Fault Address +* Register (FAR) si it is a Modified Virtual Address (MVA), NOT the Virtual +* Address (VA) as seen by the core and the programes. +*/ asmlinkage void -do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) +do_DataAbort(unsigned long mv_addr, unsigned int fsr, struct pt_regs *regs) { + unsigned long v_addr = mva_to_va(mv_addr, current->mm); /* Unmodified VA */ + pmd_t cpd = *pmd_offset(pgd_offset_k(mv_addr), mv_addr); const struct fsr_info *inf = fsr_info + (fsr & 15); #ifdef CONFIG_KGDB + #warn [JNZ] I am not sure how to handle CONFIG_KGDB with FASS yet if(kgdb_fault_expected) kgdb_handle_bus_error(); #endif - if (!inf->fn(addr, fsr, regs)) + /* The following code does the translation fault on a page to domain fault on + * a page check/conversion. Could be cleaned up probably */ + + if((pmd_domain(cpd) >= DOMAIN_START) && + !domain_active(get_dacr(), pmd_domain(cpd)) && + ((fsr & 15) == 7)) { /* 7 is translation fault on page */ + inf = fsr_info + 9; /* 9 is domain fault on page */ + } + /* + switch (fsr & 0xf) { + case 0xB: + case 0x9: + if((((fsr & 0xF0) >> 4) != pmd_domain(cpd)) && (get_dacr() != 0x15)) { + printk("** do_DataAbort: VA 0x%x MVA 0x%x fsr 0x%x %d dacr 0x%x **\n", + v_addr, mv_addr, fsr & 0xF, (fsr & 0xF0) >> 4, get_dacr()); + dump_cpd(1); + } + break; + default: + break; + } + */ + if (!inf->fn(v_addr, fsr, regs)) return; - printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at 0x%08lx\n", - inf->name, fsr, addr); + printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at va 0x%08lx mva 0x%08lx\n", + inf->name, fsr, v_addr, mv_addr); force_sig(inf->sig, current); - show_pte(current->mm, addr); + show_pte(current->mm, v_addr); #ifdef CONFIG_KGDB if(!user_mode(regs)) @@ -105,10 +254,32 @@ die_if_kernel("Oops", regs, 0); } +/* awiggins + * do_prefetchAbort: Modified to check if the reason for the abort and call + * one of the following functions: + * - do_cpd_fault(): Fault due to invalid CPD entry. + * - do_domain_fault(): Fault due to inactive domain. + * - do_translation_fault(): All other cases. + * Notes: + * - For prefetch aborts the faulting address is calculated by the kernel not + * supplied by the ARM Fault Address Register (FAR) so it is a Virtual + * Address (VA), NOT a Modified Virtual Address (MVA). + */ asmlinkage void -do_PrefetchAbort(unsigned long addr, struct pt_regs *regs) +do_PrefetchAbort(unsigned long v_addr, struct pt_regs *regs) { - do_translation_fault(addr, 0, regs); + unsigned long mv_addr = va_to_mva(v_addr, current->mm); /* MVA */ + pmd_t cpd = *pmd_offset(pgd_offset_k(mv_addr), mv_addr); + + if(pmd_none(cpd)) { + do_cpd_fault(v_addr, 0, regs); + } + else if(!domain_active(get_dacr(), pmd_domain(cpd))) { + do_domain_fault(v_addr, 0, regs); + } + else { + do_translation_fault(v_addr, 0, regs); + } } /* @@ -140,10 +311,14 @@ /* * If this page isn't present, or is already setup to * fault (ie, is old), we can safely ignore any issues. + * + * FASS: Marked as unbufferable too otherwise we could miss an update + * that is still in the write-buffer. */ - if (pte_present(entry) && pte_val(entry) & L_PTE_CACHEABLE) { + if (pte_present(entry) && pte_val(entry) & + (L_PTE_CACHEABLE | L_PTE_BUFFERABLE)) { flush_cache_page(vma, address); - pte_val(entry) &= ~L_PTE_CACHEABLE; + pte_val(entry) &= ~(L_PTE_CACHEABLE | L_PTE_BUFFERABLE); set_pte(pte, entry); flush_tlb_page(vma, address); ret = 1; @@ -163,6 +338,7 @@ return 0; } +/* awiggins: Not using this as I'm too lazy to properly patch things for now */ static void make_coherent(struct vm_area_struct *vma, unsigned long addr, struct page *page, int dirty) { @@ -211,35 +387,196 @@ flush_cache_page(vma, addr); } +/* writable_pte: Tests if a page is writable. */ +static int +writable_pte(struct vm_area_struct *vma, unsigned long address) +{ + pgd_t *pgd; + pmd_t *pmd; + pte_t *pte, entry; + + pgd = pgd_offset(vma->vm_mm, address); + if(pgd_none(*pgd)) { + return 0; + } + + pmd = pmd_offset(pgd, address); + if(pmd_none(*pmd)) { + return 0; + } + + pte = pte_offset(pmd, address); + entry = *pte; + + //printk("** writable_pte: entry = 0x%x **\n", entry); + + if(pte_present(entry)) { + //printk("** writable_pte: entry 0x%x writable %x **\n", + // entry, pte_write(entry)); + + return pte_write(entry); + } + + return 0; + +} /* writable_pte() */ + /* * Take care of architecture specific things when placing a new PTE into - * a page table, or changing an existing PTE. Basically, there are two + * a page table, or changing an existing PTE. Basically, there are three * things that we need to take care of: * * 1. If PG_dcache_dirty is set for the page, we need to ensure * that any cache entries for the kernels virtual memory * range are written back to the page. * 2. If we have multiple shared mappings of the same space in - * an object, we need to deal with the cache aliasing issues. + * an object, we need to deal with the cache synonym issues. + * 3. If the page is part of a shared vma, here we check if vm_shared_data has + * a region_struct allocated. If not a suitable one is set. * * Note that the page_table_lock will be held. + * + * FASS Notes: + * - Since we do not flush the cache on every context switch shared + * mappings that map to different virtual addresses cause synonyms in the + * cache, these synonyms independent of whether the shared mapping is within + * the same or between different address-spaces. There are two possible + * solutions: + * + * 1) Mark the shared mappings as uncachable and unbufferable. This reduces + * the performance of every access of a shared mapping, infact this is + * only needed if the mappings map to different virtual addresses. This + * is how ARM Linux currently handles shared mapping, ie synonyms, within + * the same address-space. + * + * 2) Only one shared mapping is enabled at a time. This can infact be + * extented to only one set of mappings that map the shared physical + * frame to the same virtual page are enabled at a time. The performance + * lost here is a page fault every time a disabled mapping is re-enabled + * and the synonym at different addresses are disabled. This solution is + * also independed with regard to the whether the synonym is within the + * same address-space or between address-spaces. An additional problem + * with this solution is any shared mappings within the PID relocated + * area, ie the first 32MB will not flag as synonyms when it will be if + * the sharing address-spaces have different PID's. This could be checked + * for easily enough though. + * + * I want to benchmark the performance of both solutions but for now I'll + * use 1) since it is ARM Linux's original solution. I'm also going to add + * a check to only mark the pages un(cacheable/bufferable) iff at least one + * or more mappings are writable. Otherwise the mappings will always be + * consistent. + * - The checks to use the same region are technically incorrect, if we have a + * process sharing a mapping read-only and another one sharing it read-write + * both will have read-write access to it. Checks needed to be added here to + * make sure both the access rights and the size of the shared mapping are + * the same. If they a new region must be allocated. */ void update_mmu_cache(struct vm_area_struct *vma, unsigned long addr, pte_t pte) { unsigned long pfn = pte_pfn(pte); struct page *page; + struct vm_area_struct *mpnt; + struct mm_struct *mm; + unsigned long pgoff; + int synonyms, writable; if (!pfn_valid(pfn)) return; page = pfn_to_page(pfn); - if (page->mapping) { + + if (!page->mapping) + return; + + /* region_struct updates */ + if((vma->vm_flags & VM_SHARED) && !vma->vm_sharing_data) { + + /* Check for existing region to use */ + for(mpnt = page->mapping->i_mmap_shared; mpnt; + mpnt = mpnt->vm_next_share) { + + /* If we have a shared vma of the same access type */ + if(mpnt->vm_sharing_data && + (vma->vm_start == mpnt->vm_start) && + (vma->vm_pgoff == mpnt->vm_pgoff)) { + + vma->vm_sharing_data = mpnt->vm_sharing_data; + ((struct region_struct*)vma->vm_sharing_data)->mm_count++; + + break; + } + } + + /* First vma of the shared set */ + if(!vma->vm_sharing_data) { + vma->vm_sharing_data = cpd_allocate_region(); + } + } + + { int dirty = test_and_clear_bit(PG_dcache_dirty, &page->flags); if (dirty) { unsigned long kvirt = (unsigned long)page_address(page); cpu_cache_clean_invalidate_range(kvirt, kvirt + PAGE_SIZE, 0); } + } + mm = vma->vm_mm; + pgoff = (addr - vma->vm_start) >> PAGE_SHIFT; + + writable = writable_pte(vma, addr); + synonyms = 0; + + /* Need to first check for writable synonyms */ + for(mpnt = page->mapping->i_mmap_shared; mpnt && ~writable; + mpnt = mpnt->vm_next_share) { + unsigned long off; + + /* If the shared mapping has the same region it isn't a synonym */ + if(vma->vm_sharing_data && + (vma->vm_sharing_data == mpnt->vm_sharing_data)) + continue; + + /* + * If the page isn't in this VMA, we can also ignore it. + */ + if (pgoff < mpnt->vm_pgoff) + continue; + + off = pgoff - mpnt->vm_pgoff; + if (off >= (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT) + continue; + + writable = writable_pte(mpnt, mpnt->vm_start + (off << PAGE_SHIFT)); + synonyms++; + } + + /* If there are synonyms or they are all read-only do nothing */ + if(~synonyms || ~writable) { + return; + } + + printk("** update_mmu_cache: Mark writable synonym uncachable **\n"); - make_coherent(vma, addr, page, dirty); + /* Mark all synonyms as uncachable */ + for(mpnt = page->mapping->i_mmap_shared; mpnt; + mpnt = mpnt->vm_next_share) { + unsigned long off; + + /* + * If the page isn't in this VMA, we can also ignore it. + */ + if (pgoff < mpnt->vm_pgoff) + continue; + + off = pgoff - mpnt->vm_pgoff; + if (off >= (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT) + continue; + + /* + * Ok, it is within mpnt. Fix it up. + */ + adjust_pte(mpnt, mpnt->vm_start + (off << PAGE_SHIFT)); } + adjust_pte(vma, addr); } diff -ruN linux-2.4.24-uc0-snap1/arch/arm/mm/mm-armv.c linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/mm-armv.c --- linux-2.4.24-uc0-snap1/arch/arm/mm/mm-armv.c 2004-02-17 23:52:18.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/mm-armv.c 2004-03-29 09:02:55.000000000 +1000 @@ -376,7 +376,7 @@ init_maps->physical = virt_to_phys(init_maps); init_maps->virtual = vectors_base(); init_maps->length = PAGE_SIZE; - init_maps->domain = DOMAIN_USER; + init_maps->domain = DOMAIN_KERNEL; /* awiggins for FASS */ init_maps->prot_read = 0; init_maps->prot_write = 0; init_maps->cacheable = 1; diff -ruN linux-2.4.24-uc0-snap1/arch/arm/mm/pid.c linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/pid.c --- linux-2.4.24-uc0-snap1/arch/arm/mm/pid.c 1970-01-01 10:00:00.000000000 +1000 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/mm/pid.c 2004-03-29 09:02:55.000000000 +1000 @@ -0,0 +1,263 @@ +/******************************************************************************* +* Filename: arch/arm/mm/pid.c * +* Description: ARM Process ID (PID) functions for Fast Address Space Switching * +* (FASS) in ARM Linux. * +* Created: 14/10/2001 * +* Changes: 19/02/2002 - arm_pid_allocate() modifed to create dummy vmas. * +* 29/02/2002 - Fixed arm_pid_allocate() to update the CPU PID * +* register only when the current task is the one receiving the * +* PID. * +* Copyright: (C) 2001, 2002 Adam Wiggins * +******************************************************************************** +* Notes: * +* - Even with processors that have 128 pids we might want to limit the * +* relocation to the first 64 so we have seperate area's for the text,data, * +* bss, stack, etc relocation and arch_get_unmapped_area() area's for * +* non-MAP_FIXED mmap()'s * +* - This is evily hardcoded anyway, really some arch specific mm code should * +* do the initialision. * +******************************************************************************** +* This program is free software; you can redistribute it and/or modify * +* it under the terms of the GNU General Public Licence version 2 as * +* published by the Free Software Foundation. * +*******************************************************************************/ + +#include +#include + +#include + +/********** +* Globals * +**********/ + +/* The next PID to be allocated. Hack for now */ +static int next_pid_num = ARMPID_START; + +/* Array of pid_structs for StrongARM's 64 pids */ +static struct pid_struct pids[ARMPID_MAX] = { + {number:0, mm_count:0}, + {number:1, mm_count:0}, + {number:2, mm_count:0}, + {number:3, mm_count:0}, + {number:4, mm_count:0}, + {number:5, mm_count:0}, + {number:6, mm_count:0}, + {number:7, mm_count:0}, + {number:8, mm_count:0}, + {number:9, mm_count:0}, + {number:10, mm_count:0}, + {number:11, mm_count:0}, + {number:12, mm_count:0}, + {number:13, mm_count:0}, + {number:14, mm_count:0}, + {number:15, mm_count:0}, + {number:16, mm_count:0}, + {number:17, mm_count:0}, + {number:18, mm_count:0}, + {number:19, mm_count:0}, + {number:20, mm_count:0}, + {number:21, mm_count:0}, + {number:22, mm_count:0}, + {number:23, mm_count:0}, + {number:24, mm_count:0}, + {number:25, mm_count:0}, + {number:26, mm_count:0}, + {number:27, mm_count:0}, + {number:28, mm_count:0}, + {number:29, mm_count:0}, + {number:30, mm_count:0}, + {number:31, mm_count:0}, + {number:32, mm_count:0}, + {number:33, mm_count:0}, + {number:34, mm_count:0}, + {number:35, mm_count:0}, + {number:36, mm_count:0}, + {number:37, mm_count:0}, + {number:38, mm_count:0}, + {number:39, mm_count:0}, + {number:40, mm_count:0}, + {number:41, mm_count:0}, + {number:42, mm_count:0}, + {number:43, mm_count:0}, + {number:44, mm_count:0}, + {number:45, mm_count:0}, + {number:46, mm_count:0}, + {number:47, mm_count:0}, + {number:48, mm_count:0}, + {number:49, mm_count:0}, + {number:50, mm_count:0}, + {number:51, mm_count:0}, + {number:52, mm_count:0}, + {number:53, mm_count:0}, + {number:54, mm_count:0}, + {number:55, mm_count:0}, + {number:56, mm_count:0}, + {number:57, mm_count:0}, + {number:58, mm_count:0}, + {number:59, mm_count:0}, + {number:60, mm_count:0}, + {number:61, mm_count:0}, + {number:62, mm_count:0}, + {number:63, mm_count:0}, +}; + +/************ +* Functions * +************/ + +/* arm_pid_allocate: Allocates an ARM PID to 'mm_p' address-space. If the + * address-space is marked as relocatable. The location of the stack dictates + * this relocatability. If the stack is within the 32MB relocation area the + * address-space will be allocated a ARM PID, otherwise the zero PID, no + * relocation is allocated.If a pid is selected two dummy, that is + * unaccessable, vm_areas are added. The first covers the 32MB relocation + * synonym to stop that area from being mmap()'able. The second tries to + * protection from the stack overrunning into the bss segment or visa versa. + * These dumma va_areas are not copied by fork(). In this case a new PID will + * be allocated as well as the new dummy vm_areas to go with it. + * Notes: + * - This function is only called by arch/arm/mmu_context.c:init_new_context() + * which is in turn called by fs/exec.c:exec_mmap() and + * kernel/fork.c:copy_mm() + * - fs/exec.c:exec_mmap() seems to have no way of indicating to + * init_new_context()/allocate_pid() any information to decide if a newly + * exec'ed process should have an ARM PID allocated or not. We need some way + * to indicate to the STACK_TOP macro whether a task should be relocated or + * not. + * - Simply allocates ARM PIDs from 1 through to 63. A loop selects a PID with + * the least count, that is least number of processes allocated to it. This + * should be reexamined to see if a more appropriate method is avaliable. + * ie, we might want to make sure on fork() that the new childs ARM PID is + * different to the parents one. + */ +void +arm_pid_allocate(struct task_struct* task_p, struct mm_struct* mm_p) +{ + int i; + int pid_num; + unsigned long vm_flags; + struct vm_area_struct* vma_p; + + //printk("** arm_pid_allocate: mm_p 0x%x vmas %d **\n", mm_p, mm_p->map_count); + //dump_vma(mm_p); + +#if PID_ENABLE + + if(mm_p->start_stack > ARMPID_TASK_SIZE) { /* No pid allocated to parent */ + pid_num = 0; + + goto allocate; /* PID remains zero */ + } + +#else + + goto allocate; /* PID remains zero */ + +#endif + + /* Select an ARM PID for allocation */ + //pid_num = next_pid_num; + //next_pid_num = (next_pid_num >= (ARMPID_MAX - 1) ? + // ARMPID_START : next_pid_num + 1); + + for(i = pid_num = 1; pids[pid_num].mm_count && (i < ARMPID_MAX); i++) { + if(pids[pid_num].mm_count > pids[i].mm_count) { + pid_num = i; + } + } + + //printk("** arm_pid_allocate: about to mmap dummy regions **\n"); + + /* Map 4KB dummy vma into area between stack and bss. */ + vm_flags = VM_DONTEXPAND | VM_DONTCOPY; /* Stops fork copying */ + + vma_p = kmem_cache_alloc(vm_area_cachep, SLAB_KERNEL); + + assert(vma_p); + + down_write(&mm_p->mmap_sem); + vma_p->vm_mm = mm_p; + vma_p->vm_start = ARMPID_BRK_LIMIT; + vma_p->vm_end = vma_p->vm_start + PAGE_SIZE; + vma_p->vm_flags = vm_flags; + vma_p->vm_page_prot = protection_map[vm_flags & 0x0f]; + vma_p->vm_ops = NULL; + vma_p->vm_pgoff = 0; + vma_p->vm_file = NULL; + vma_p->vm_private_data = NULL; + vma_p->vm_sharing_data = NULL; + vma_p->vm_raend = 0; + + insert_vm_struct(mm_p, vma_p); + up_write(&mm_p->mmap_sem); + + mm_p->total_vm += PAGE_SIZE >> PAGE_SHIFT; /* Add a page */ + + + //printk("** arm_pid_allocate: Managed first mmap at 0x%x **\n", + // ARMPID_TASK_SIZE - MEGABYTE(1)); + + /* Map 32MB dummy vma into relocation point of addres-space */ + vm_flags = VM_DONTEXPAND | VM_DONTCOPY; /* Stops fork copying */ + + vma_p = kmem_cache_alloc(vm_area_cachep, SLAB_KERNEL); + + assert(vma_p); + + down_write(&mm_p->mmap_sem); + vma_p->vm_mm = mm_p; + vma_p->vm_start = PIDNUM_TO_PID(pid_num); + vma_p->vm_end = vma_p->vm_start + ARMPID_TASK_SIZE; + vma_p->vm_flags = vm_flags; + vma_p->vm_page_prot = protection_map[vm_flags & 0x0f]; + vma_p->vm_ops = NULL; + vma_p->vm_pgoff = 0; + vma_p->vm_file = NULL; + vma_p->vm_private_data = NULL; + vma_p->vm_sharing_data = NULL; + vma_p->vm_raend = 0; + + insert_vm_struct(mm_p, vma_p); + up_write(&mm_p->mmap_sem); + + mm_p->total_vm += ARMPID_TASK_SIZE >> PAGE_SIZE; + + //printk("** arm_pid_allocate: Managed second mmap at 0x%x**\n", + // PIDNUM_TO_PID(pid_num)); + + + /* Allocate ARM PID to address-space */ + allocate: + + mm_p->context.pid_p = pids + pid_num; + mm_p->context.pid = PIDNUM_TO_PID(pid_num); + pids[pid_num].mm_count++; + + if(current == task_p) { /* If this task is getting the new mm set pid */ + + //printk("** arm_pid_allocate: setting cpu pid **\n"); + + set_armpid(mm_p->context.pid); + } + + //printk("** arm_pid_allocate: allocated ARM PID %d to mm_p 0x%x**\n", + // PID_TO_PIDNUM(get_armpid()), mm_p); + //dump_vma(mm_p); + +} /* arm_pid_allocate() */ + +/* arm_pid_unallocate: Simply deallocates an ARM PID from an address space. + */ +void +arm_pid_unallocate(struct pid_struct* pid_p) +{ + assert(pid_p->mm_count > 0); + //printk("** arm_unallocate_pid: unallocating ARM PID %d **\n", + // PID_TO_PIDNUM(get_armpid())); + + pid_p->mm_count--; + + set_armpid(0x0); + +} /* arm_pid_unallocate() */ diff -ruN linux-2.4.24-uc0-snap1/arch/arm/tools/getconstants.c linux-2.4.24-uc0-snap1-fass0/arch/arm/tools/getconstants.c --- linux-2.4.24-uc0-snap1/arch/arm/tools/getconstants.c 2004-02-17 23:52:18.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/arch/arm/tools/getconstants.c 2004-03-29 09:07:36.000000000 +1000 @@ -26,6 +26,7 @@ #define OFF_TSK(n) (unsigned long)&(((struct task_struct *)0)->n) #define BOFF_TSK(n) ((unsigned long)&(((struct task_struct *)0)->n))+1 +#define OFF_MM(n) (unsigned long)&(((struct mm_struct *)0)->n) #define DEFN(name,off) asm("\n#define "name" %0" :: "I" (off)) @@ -42,7 +43,7 @@ DEFN("TSS_FPESAVE", OFF_TSK(thread.fpstate.soft.save)); #ifdef CONFIG_CPU_32 -DEFN("TSS_DOMAIN", OFF_TSK(thread.domain)); +DEFN("TSS_DACR", OFF_TSK(thread.dacr)); DEFN("HPTE_TYPE_SMALL", PTE_TYPE_SMALL); DEFN("HPTE_AP_READ", PTE_AP_READ); @@ -56,6 +57,10 @@ DEFN("LPTE_WRITE", L_PTE_WRITE); DEFN("LPTE_EXEC", L_PTE_EXEC); DEFN("LPTE_DIRTY", L_PTE_DIRTY); + +DEFN("TSK_MM", OFF_TSK(mm)); +DEFN("MM_CONTEXT_DACR", OFF_MM(context.dacr)); +DEFN("MM_CONTEXT_PID", OFF_MM(context.pid)); #endif #ifdef CONFIG_CPU_26 diff -ruN linux-2.4.24-uc0-snap1/fs/binfmt_aout.c linux-2.4.24-uc0-snap1-fass0/fs/binfmt_aout.c --- linux-2.4.24-uc0-snap1/fs/binfmt_aout.c 2004-02-17 23:53:42.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/fs/binfmt_aout.c 2004-03-29 09:02:55.000000000 +1000 @@ -304,7 +304,12 @@ (current->mm->start_brk = N_BSSADDR(ex)); current->mm->rss = 0; - current->mm->mmap = NULL; + /* Why is this here? It breaks my code by losing the reference to the + * dummy VMAs required by ARM FASS and the mm_struct is already zeroed + * out by kernel/fork.c:mm_alloc() anyway. Commented it out for now, + * needs to be resolved later. + */ + /*current->mm->mmap = NULL; FASS hack */ compute_creds(bprm); current->flags &= ~PF_FORKNOEXEC; #ifdef __sparc__ diff -ruN linux-2.4.24-uc0-snap1/fs/binfmt_elf.c linux-2.4.24-uc0-snap1-fass0/fs/binfmt_elf.c --- linux-2.4.24-uc0-snap1/fs/binfmt_elf.c 2004-02-17 23:53:42.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/fs/binfmt_elf.c 2004-03-29 09:02:55.000000000 +1000 @@ -617,7 +617,12 @@ current->mm->start_data = 0; current->mm->end_data = 0; current->mm->end_code = 0; - current->mm->mmap = NULL; + /* Why is this here? It breaks my code by losing the reference to the + * dummy VMAs required by ARM FASS and the mm_struct is already zeroed + * out by kernel/fork.c:mm_alloc() anyway. Commented it out for now, + * needs to be resolved later. + */ + /*current->mm->mmap = NULL; FASS hack */ current->flags &= ~PF_FORKNOEXEC; elf_entry = (unsigned long) elf_ex.e_entry; diff -ruN linux-2.4.24-uc0-snap1/fs/exec.c linux-2.4.24-uc0-snap1-fass0/fs/exec.c --- linux-2.4.24-uc0-snap1/fs/exec.c 2004-02-17 23:53:42.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/fs/exec.c 2004-03-29 09:02:55.000000000 +1000 @@ -365,6 +365,7 @@ mpnt->vm_pgoff = 0; mpnt->vm_file = NULL; mpnt->vm_private_data = (void *) 0; + mpnt->vm_sharing_data = NULL; insert_vm_struct(current->mm, mpnt); current->mm->total_vm = (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT; } @@ -444,7 +445,12 @@ mm = mm_alloc(); if (mm) { - struct mm_struct *active_mm; + + struct mm_struct *active_mm; + + //printk("exec_mmap: current 0x%x old mm_p 0x%x new mm_p 0x%x **\n", + // current, old_mm, mm); /* awiggins */ + if (init_new_context(current, mm)) { mmdrop(mm); @@ -922,6 +928,9 @@ int retval; int i; + //printk("** exec: current 0x%x mm_p 0x%x **\n", + // current, current->mm); /* awiggins */ + file = open_exec(filename); retval = PTR_ERR(file); diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/a.out.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/a.out.h --- linux-2.4.24-uc0-snap1/include/asm-arm/a.out.h 2004-02-17 23:53:48.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/a.out.h 2004-03-29 09:02:55.000000000 +1000 @@ -28,8 +28,18 @@ #define M_ARM 103 #ifdef __KERNEL__ -#define STACK_TOP ((current->personality == PER_LINUX_32BIT) ? \ - TASK_SIZE : TASK_SIZE_26) + +/*#define STACK_TOP ((current->personality == PER_LINUX_32BIT) ? \ + TASK_SIZE : TASK_SIZE_26)*/ + +/* FASS sets STACK_TOP to top of PID relocated memory, ie 32MB. In all reality + * there should be some heuristic for selecting of the old STACK_TOP or + * 32MB should be set or not. This basically boils down to if a non-zero PID + * is going to be used. Unfortunately when the stack is set in exec() that is + * too difficult to work out. + */ +#define STACK_TOP ARMPID_TASK_SIZE + #endif #ifndef LIBRARY_START_TEXT diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/elf.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/elf.h --- linux-2.4.24-uc0-snap1/include/asm-arm/elf.h 2004-02-17 23:53:49.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/elf.h 2004-03-29 09:02:55.000000000 +1000 @@ -43,7 +43,7 @@ the loader. We need to make sure that it is out of the way of the program that it will "exec", and that there is sufficient room for the brk. */ -#define ELF_ET_DYN_BASE (2 * TASK_SIZE / 3) +#define ELF_ET_DYN_BASE CPD_UNMAPPED_BASE /* When the program starts, a1 contains a pointer to a function to be registered with atexit, as per the SVR4 ABI. A value of 0 means we diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/mmu.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/mmu.h --- linux-2.4.24-uc0-snap1/include/asm-arm/mmu.h 2004-02-17 23:53:50.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/mmu.h 2004-03-29 09:02:55.000000000 +1000 @@ -1,9 +1,43 @@ +/* include/asm-arm/mmu.h + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * 20-Jun-2001 Modified by Adam Wiggins + * for FASS. + * 11-Oct-2001 Added PID field for PID register relocation support. + * 26-Apr-2002 Added domain_timestamp field allow lazy clearing of the + * DACR. This is required to keep track of clean domains + * and to disable accessed to unallocated domains. + * 15-Sep-2003 Modified by John Zaitseff + * for Linux kernel 2.4.21-rmk1 + */ + #ifndef __ARM_MMU_H #define __ARM_MMU_H -/* - * The ARM doesn't have a mmu context - */ -typedef struct { } mm_context_t; +/*********************** +* Forward declerations * +***********************/ + +/* include/asm-arm/proc-armv/cpd.h */ +struct domain_struct; + +/* include/asm-arm/proc-armv/pid.h */ +struct pid_struct; + +/* Address-space context */ +struct fass_struct{ + unsigned int pid; /* Process ID register */ + unsigned int dacr; /* Domain Access Control Register */ + unsigned long long domain_timestamp; /* Last DACR clearing to check domains */ + + struct pid_struct* pid_p; /* PID associated with address-space */ + struct region_struct* region_p; /* Address-space's private region */ +}; + +typedef struct fass_struct mm_context_t; + -#endif +#endif /* !__ARM_MMU_H */ diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/mmu_context.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/mmu_context.h --- linux-2.4.24-uc0-snap1/include/asm-arm/mmu_context.h 2004-02-17 23:53:50.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/mmu_context.h 2004-03-29 09:02:55.000000000 +1000 @@ -9,6 +9,7 @@ * * Changelog: * 27-06-1996 RMK Created + * 20-06-2001 AGW Defined (destroy/init_new)_context for FASS. */ #ifndef __ASM_ARM_MMU_CONTEXT_H #define __ASM_ARM_MMU_CONTEXT_H @@ -16,9 +17,42 @@ #include #include #include +#include +#include -#define destroy_context(mm) do { } while(0) -#define init_new_context(tsk,mm) 0 +static inline void +destroy_context(struct mm_struct* mm_p) +{ + //printk("** destroy_context: current 0x%x mm_p 0x%x **\n", current, mm_p); + + cpd_unallocate_region(mm_p->context.region_p); + + if(mm_p->context.pid_p) { + arm_pid_unallocate(mm_p->context.pid_p); + } + +} /* destroy_context() */ + +/* Do these values need to be zeroed? It seemed not to work if they weren't */ +static inline int +init_new_context(struct task_struct* task_p, struct mm_struct* mm_p) +{ + //printk("** init_new_context: task_p 0x%x mm_p 0x%x **\n", task_p, mm_p); + + if((mm_p->context.region_p = cpd_allocate_region()) == NULL) { + return -1; /* Is this right? */ + } + + /* No domain or ARM PID yet allocated */ + mm_p->context.pid = 0; + mm_p->context.pid_p = NULL; + mm_p->context.dacr = 0; + + arm_pid_allocate(task_p, mm_p); + + return 0; + +} /* init_new_context() */ /* * This is called when "tsk" is about to enter lazy TLB mode. @@ -43,7 +77,7 @@ struct task_struct *tsk, unsigned int cpu) { if (prev != next) - cpu_switch_mm(next->pgd, tsk); + cpd_switch_mm(next); } #define activate_mm(prev, next) \ diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/pgtable.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/pgtable.h --- linux-2.4.24-uc0-snap1/include/asm-arm/pgtable.h 2004-02-17 23:53:50.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/pgtable.h 2004-03-29 09:02:55.000000000 +1000 @@ -160,6 +160,9 @@ #include +/* FASS defines arch_get_unmapped_area */ +#define HAVE_ARCH_UNMAPPED_AREA + extern void pgtable_cache_init(void); /* diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/cache.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/cache.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/cache.h 2004-02-17 23:53:52.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/cache.h 2004-03-29 09:02:55.000000000 +1000 @@ -32,31 +32,29 @@ */ #define flush_cache_all() \ do { \ - cpu_cache_clean_invalidate_all(); \ + cpd_cache_flush_all(); \ } while (0) /* This is always called for current->mm */ #define flush_cache_mm(_mm) \ do { \ if ((_mm) == current->active_mm) \ - cpu_cache_clean_invalidate_all(); \ + cpd_cache_flush_mm((_mm)); \ } while (0) #define flush_cache_range(_mm,_start,_end) \ do { \ if ((_mm) == current->active_mm) \ - cpu_cache_clean_invalidate_range((_start) & PAGE_MASK, \ - PAGE_ALIGN(_end), 1); \ + cpd_cache_flush_range((_mm), (_start) & PAGE_MASK, \ + PAGE_ALIGN(_end)); \ } while (0) #define flush_cache_page(_vma,_vmaddr) \ do { \ if ((_vma)->vm_mm == current->active_mm) { \ unsigned long _addr = (_vmaddr) & PAGE_MASK; \ - cpu_cache_clean_invalidate_range(_addr, \ - _addr + PAGE_SIZE, \ - ((_vma)->vm_flags & VM_EXEC)); \ - } \ + cpd_cache_flush_page((_vma), _addr); \ + } \ } while (0) /* @@ -97,6 +95,9 @@ /* * D cache only + * + * Note: FASS doesn't touch these since they seem to be for keeping kernel memory + * consistent with the caches. */ #define invalidate_dcache_range(_s,_e) cpu_dcache_invalidate_range((_s),(_e)) @@ -193,7 +194,7 @@ */ #define flush_tlb_all() \ do { \ - cpu_tlb_invalidate_all(); \ + cpd_tlb_flush_all(); \ } while (0) /* @@ -206,7 +207,7 @@ #define flush_tlb_mm(_mm) \ do { \ if ((_mm) == current->active_mm) \ - cpu_tlb_invalidate_all(); \ + cpd_tlb_flush_mm((_mm)); \ } while (0) /* @@ -217,7 +218,7 @@ #define flush_tlb_range(_mm,_start,_end) \ do { \ if ((_mm) == current->active_mm) \ - cpu_tlb_invalidate_range((_start), (_end)); \ + cpd_tlb_flush_range((_mm), (_start), (_end)); \ } while (0) /* @@ -226,8 +227,7 @@ #define flush_tlb_page(_vma,_page) \ do { \ if ((_vma)->vm_mm == current->active_mm) \ - cpu_tlb_invalidate_page((_page), \ - ((_vma)->vm_flags & VM_EXEC)); \ + cpd_tlb_flush_page((_vma), (_page)); \ } while (0) /* diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/cpd.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/cpd.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/cpd.h 1970-01-01 10:00:00.000000000 +1000 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/cpd.h 2004-03-29 09:02:55.000000000 +1000 @@ -0,0 +1,308 @@ +/******************************************************************************* +* Filename: include/asm-arm/proc-armv/cpd.h * +* Description: Caching Page Directory (CPD) includes for Fast Address Space * +* Switching (FASS) in ARM Linux. * +* Created: 20/06/2001 * +* Changes: 19/02/2002 - Added helper macros. * +* 22/02/2002 - cpd_struct updated to include doubly linked list * +* of cpd_structs allocated to the same domain. * +* 19/03/2002 - Start of modifications to support shared domains. * +* 26/04/2002 - Change per address-space domain disabling, via the * +* DACR, to be lazy rather then diabling all domains on a context * +* switch. Now all domains only diabled if a cache/tlb clean event * +* occured since the address-space last ran or if a domain was * +* unallocated. * +* 28/04/2002 - Added macro's for clock domain recycling algorithm.* +* Copyright: (C) 2001, 2002 Adam Wiggins * +******************************************************************************** +* This program is free software; you can redistribute it and/or modify * +* it under the terms of teh GNU General Public License version 2 as * +* published by the Free Software Foundation. * +*******************************************************************************/ + +#ifndef __ASM_PROC_CPD_H +#define __ASM_PROC_CPD_H + +#include +#include + +/****************** +* Benchmark Flags * +******************/ + +#define CLOCK_RECYCLE 0 /* Round-robin domain recycling */ + +#define PID_ENABLE 1 /* PID relocation enabling/disabling */ + +#define LAZY_DISABLE 1 /* Lazy vs strick domain disabling via DACR */ + +#define ALWAYS_DIRTY 0 /* Always mark tlb/caches as incoherent */ + +/********** +* Defines * +**********/ + +#define HAVE_ARCH_VM_SHARING_DATA + +/* Base address for mmap() allocations */ +#define CPD_UNMAPPED_BASE 0x80000000 /* Evilly hardcoded to 2GB */ + +/********* +* Macros * +*********/ + +/* Set to enable FASS specific debugging options, yes there is probably a better + * way to do this. */ +#if 1 +#define assert(expr) if(!(expr)) BUG() +#else +#define assert(expr) +#endif + +/* Some handy macros */ + +#define MEGABYTE(n) ((n) << 20) + +#define MASK_MEGABYTE(addr, n) ((addr) & ~(MEGABYTE(n) - 1)) + +#define CPD_P_TO_INDEX(cpd_p) (((unsigned long)(cpd_p) >> 2) & \ + ((1 << 12) - 1)) /* I know this is ugly */ + +#define STATS_P_TO_INDEX(cpd_stats_p) ((((unsigned long)(cpd_stats_p)) - \ + ((unsigned long)(cpd_stats))) / \ + sizeof(struct cpd_struct)) + +/* Is a domain/mm dirty with respect to the TLBs/Caches? */ + +#if ALWAYS_DIRTY + +#define cpd_is_domain_coherent(domain) 0 +#define cpd_is_domain_tlb_coherent(domain) 0 +#define cpd_is_domain_cache_coherent(domain) 0 +#define cpd_is_domain_dcache_coherent(domain) 0 +#define cpd_is_domain_icache_coherent(domain) 0 + +#define cpd_is_mm_coherent(mm_p) 0 +#define cpd_is_mm_tlb_coherent(mm_p) 0 +#define cpd_is_mm_cache_coherent(mm_p) 0 +#define cpd_is_mm_dcache_coherent(mm_p) 0 +#define cpd_is_mm_icache_coherent(mm_p) 0 + +#else + +#define cpd_is_domain_coherent(domain) \ + (cpd_is_domain_tlb_coherent(domain) && \ + cpd_is_domain_cache_coherent(domain)) +#define cpd_is_domain_tlb_coherent(domain) \ + !domain_active(domain_tlb_coherence, (domain)) +#define cpd_is_domain_cache_coherent(domain) \ + (cpd_is_domain_dcache_coherent(domain) && \ + cpd_is_domain_icache_coherent(domain)) +#define cpd_is_domain_dcache_coherent(domain) \ + !domain_active(domain_dcache_coherence, (domain)) +#define cpd_is_domain_icache_coherent(domain) \ + !domain_active(domain_icache_coherence, (domain)) + +#define cpd_is_mm_coherent(mm_p) \ + (cpd_is_mm_tlb_coherent(mm_p) && cpd_is_mm_cache_coherent(mm_p)) +#define cpd_is_mm_tlb_coherent(mm_p) \ + (!((mm_p)->context.dacr & domain_tlb_coherence & \ + SYSTEM_DOMAINS_MASK)) +#define cpd_is_mm_cache_coherent(mm_p) \ + (cpd_is_mm_dcache_coherent((mm_p)) && \ + cpd_is_mm_icache_coherent((mm_p))) +#define cpd_is_mm_dcache_coherent(mm_p) \ + (!((mm_p)->context.dacr & domain_dcache_coherence & \ + SYSTEM_DOMAINS_MASK)) +#define cpd_is_mm_icache_coherent(mm_p) \ + (!((mm_p)->context.dacr & domain_icache_coherence & \ + SYSTEM_DOMAINS_MASK)) + +#endif + +#if CLOCK_RECYCLE + +#define cpd_domain_touched(domain) \ + (domain_active(domain_touched, (domain))) + +#define cpd_set_domain_touched(domain) \ + (domain_touched |= domain_val((domain), DOMAIN_CLIENT)) + +#define cpd_clear_domain_touched(domain) \ + (domain_touched &= ~(domain_val((domain), DOMAIN_MANAGER))) + +#endif + +/* Clean the TLBs/Caches */ + +#define cpd_tlb_clean() \ +do { \ + /*printk("** cpd_tlb_clean() **\n");*/ \ + cpu_tlb_invalidate_all(); \ + set_dacr(current->thread.dacr); /* To keep track of dirty domains */ \ + domain_clock++; /* as above */ \ + domain_tlb_coherence = 0; \ +} while(0) + +#define cpd_cache_clean() \ +do { \ + /*printk("** cpd_cache_clean() **\n");*/ \ + cpu_cache_clean_invalidate_all(); \ + set_dacr(current->thread.dacr); /* To keep track of dirty domains */ \ + domain_clock++; /* as above */ \ + domain_dcache_coherence = 0; \ + domain_icache_coherence = 0; \ +} while(0) + +#define cpd_dcache_clean() \ +do { \ + cpu_dcache_clean_invalidate_all(); \ + set_dacr(current->thread.dacr); /* To keep track of dirty domains */ \ + domain_clock++; /* as above */ \ + domain_dcache_coherence = 0; \ +} while(0) + +/*********** +* Typedefs * +***********/ + +struct domain_struct; /* Dummy prototype */ + +/* Region context */ +struct region_struct { + int mm_count; /* Number of address-spaces using region */ + struct domain_struct* domain_p; /* Domain this region is tagged with */ + struct region_struct* next; /* Next region using this domain */ +}; + +/* Domain context */ +struct domain_struct { + unsigned int number; /* Domain number */ + unsigned int cpd_count; /* CPD entries associated with this domain */ + struct cpd_struct* cpd_head; /* List of CPD entries with this domain */ + struct region_struct* region_p; /* Region that owns this domain currently */ +}; + +/* CPD information + * Notes: + * - Number of collisions can be maintained in cpd_load. + * - Number of mm_struct's using this cpd entry, don't know how to calculate. + */ +struct cpd_struct { + unsigned int collisions; /* Number collisions on this entry */ + unsigned int mm_count; /* Number of mm_struct's using this cpd entry */ + struct cpd_struct* cpd_next; /* Next CPD entry with the same domain */ + struct cpd_struct* cpd_prev; /* Previous CPD entry with the same domain */ +}; + +/********** +* Externs * +**********/ + +extern struct domain_struct domains[DOMAIN_MAX]; +extern struct cpd_struct cpd_stats[USER_PTRS_PER_PGD]; + +/******************* +* Inline Functions * +*******************/ + + +static inline void +cpd_stats_add(struct domain_struct* domain_p, struct cpd_struct* cpd_stats_p) +{ + assert(domain_p); + assert(cpd_stats_p); + //printk("** cpd_stats_add: cpd_struct_p 0x%x **\n", (unsigned int)cpd_stats_p); + //dump_cpd_stats(domain_p->number); + assert(cpd_stats_p != domain_p->cpd_head); + + cpd_stats_p->cpd_next = domain_p->cpd_head; + cpd_stats_p->cpd_prev = NULL; + + if(domain_p->cpd_head) { + domain_p->cpd_head->cpd_prev = cpd_stats_p; + } + domain_p->cpd_head = cpd_stats_p; + + //dump_cpd_stats(domain_p->number); + +} /* cpd_stats_add() */ + +/* cpd_stats_del: Simply removes the cpd_stat from the domain_p's list */ +static inline void +cpd_stats_del(int domain, struct cpd_struct* cpd_stats_p) +{ + assert(cpd_stats_p); + //printk("** cpd_stats_del: cpd_struct_p 0x%x prev 0x%x next 0x%x **\n", + // (unsigned int)cpd_stats_p, (unsigned int)cpd_stats_p->cpd_prev, + // (unsigned int)cpd_stats_p->cpd_next); + + if(domains[domain].cpd_head == cpd_stats_p) { + domains[domain].cpd_head = cpd_stats_p->cpd_next; + } + + /* De-link from list */ + if(cpd_stats_p->cpd_prev) { + cpd_stats_p->cpd_prev->cpd_next = cpd_stats_p->cpd_next; + } + + if(cpd_stats_p->cpd_next) { + cpd_stats_p->cpd_next->cpd_prev = cpd_stats_p->cpd_prev; + } + +} /* cpd_stats_del() */ + +/*********************** +* Founction Prototypes * +***********************/ + +void cpd_switch_mm(struct mm_struct* mm_p); + +/* CPD Manipulation */ +void cpd_load(pmd_t pmd, unsigned long mv_addr); + +/* Region Manipulation */ +struct region_struct* cpd_allocate_region(void); +void cpd_unallocate_region(struct region_struct* region_p); + +/* Domain Manipulation */ +int cpd_get_active_domain(struct mm_struct* mm_p, unsigned int v_addr); +struct domain_struct* cpd_get_domain(struct mm_struct* mm_p, + unsigned int v_addr); +struct domain_struct* cpd_allocate_domain(struct region_struct* region_p); +void cpd_unallocate_domain(struct domain_struct* domain_p); + +/* mmap address allocation */ + +inline unsigned long cpd_next_area(unsigned long given_addr); +unsigned long cpd_get_shared_area(struct file *file_p, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags); +unsigned long cpd_get_unique_area(struct file *file_p, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags); +unsigned long cpd_get_private_area(struct file *file_p, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags); +unsigned long cpd_get_alloced_area(struct file *file_p, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags); + +/* TLB/CPD/PageTable Coherence */ +void cpd_set_pmd(pmd_t* pmd_p, pmd_t pmd); + +void cpd_tlb_flush_all(void); +void cpd_tlb_flush_mm(struct mm_struct* mm_p); +void cpd_tlb_flush_range(struct mm_struct* mm_p, + unsigned long va_start, unsigned long va_end); +void cpd_tlb_flush_page(struct vm_area_struct* vma_p, unsigned long va_page); + +/* Cache Coherence */ +void cpd_cache_flush_all(void); +void cpd_cache_flush_mm(struct mm_struct* mm_p); +void cpd_cache_flush_range(struct mm_struct* mm_p, + unsigned long va_start, unsigned long va_end); +void cpd_cache_flush_page(struct vm_area_struct* vma_p, unsigned long va_page); + + +#endif /* !__ASM_PROC_CPD_H */ diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/domain.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/domain.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/domain.h 2004-02-17 23:53:52.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/domain.h 2004-03-29 09:02:55.000000000 +1000 @@ -3,10 +3,14 @@ * * Copyright (C) 1999 Russell King. * + * 20-Jun-2001 Modified by Adam Wiggins + * for FASS. + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. */ + #ifndef __ASM_PROC_DOMAIN_H #define __ASM_PROC_DOMAIN_H @@ -16,12 +20,37 @@ * DOMAIN_IO - domain 2 includes all IO only * DOMAIN_KERNEL - domain 1 includes all kernel memory only * DOMAIN_USER - domain 0 includes all user memory only + * + * FASS Changes + * + * To support debugging the zero domain is unallocated and domains 3-15 are + * used for the USER while 1,2 are used by the KERNEL. Really only 0 should be + * used by the KERNEL and 1-15 by USER. */ -#define DOMAIN_USER 0 +#define DOMAIN_USER 0 /* Don't want user pg_dirs to be tagged */ #define DOMAIN_KERNEL 1 #define DOMAIN_TABLE 1 #define DOMAIN_IO 2 +#define DOMAIN_MAX 16 +#define DOMAIN_END (DOMAIN_MAX - 1) +#define DOMAIN_START 3 + +#define DOMAIN_USER_MASK (domain_val(DOMAIN_IO, DOMAIN_MANAGER) | \ + domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) | \ + domain_val(DOMAIN_USER, DOMAIN_NOACCESS)) + +/* [JNZ] The following is Adam's original definition: +#define SYSTEM_DOMAINS_MASK (~(domain_val(DOMAIN_USER, DOMAIN_MANAGER) && \ + domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) && \ + domain_val(DOMAIN_IO, DOMAIN_MANAGER))) +*/ +#define SYSTEM_DOMAINS_MASK (~(domain_val(DOMAIN_USER, DOMAIN_MANAGER) | \ + domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) | \ + domain_val(DOMAIN_IO, DOMAIN_MANAGER))) + + + /* * Domain types */ @@ -29,22 +58,91 @@ #define DOMAIN_CLIENT 1 #define DOMAIN_MANAGER 3 +/* Domain dirty indicators */ + +#define DIRTY 0xFFFFFFFF +#define CLEAN 0x00000000 + +extern unsigned long domain_tlb_coherence; +extern unsigned long domain_dcache_coherence; +extern unsigned long domain_icache_coherence; + +extern unsigned long domain_touched; /* Used by domain recycling clock algo */ + #define domain_val(dom,type) ((type) << 2*(dom)) -#define set_domain(x) \ - do { \ - __asm__ __volatile__( \ - "mcr p15, 0, %0, c3, c0 @ set domain" \ - : : "r" (x)); \ - } while (0) +/* Sets the CPU's Domain Access Control Register DACR */ +static inline void +set_dacr(unsigned long dacr) +{ + domain_tlb_coherence |= dacr; + domain_dcache_coherence |= dacr; + domain_icache_coherence |= dacr; + + //printk("** set_dacr: dacr 0x%x **\n", dacr); + + __asm__ __volatile__( + "mcr p15, 0, %0, c3, c0 @ set cpu dacr" + : : "r" (dacr)); + +} /* set_dacr() */ + +/* Returns the state of the CPU's DACR */ +static inline unsigned long +get_dacr(void) +{ + unsigned long dacr; + + __asm__ __volatile__( + "mrc p15, 0, %0, c3, c0 @ Get domain" + : "=&r" (dacr) : ); + + return dacr; + +} /* get_dacr() */ + +/* This macro updates the threads DACR context from task_p's mm context */ + +#if CLOCK_RECYCLE + +#define update_dacr(domain) \ +do{ \ + unsigned long dacr; \ + \ + current->mm->context.dacr |= domain_val(domain, DOMAIN_CLIENT); /* Added */ \ + dacr = current->thread.dacr | current->mm->context.dacr; \ + domain_touched |= dacr; \ + set_dacr(dacr); \ +} while(0) + +#else + +#define update_dacr(domain) \ +do{ \ + unsigned long dacr; \ + \ + current->mm->context.dacr |= domain_val(domain, DOMAIN_CLIENT); /* Added */ \ + dacr = current->thread.dacr | current->mm->context.dacr; \ + set_dacr(dacr); \ +} while(0) + + +#endif + +/* Return the domain of page middle diretory (really page directory) entry */ +#define pmd_domain(pmd) ((pmd_val(pmd) >> 5) & (DOMAIN_MAX - 1)) + +/* Tests if a domain is active in the given DACR */ +#define domain_active(dacr, domain) (((dacr) >> (2*(domain))) & DOMAIN_MANAGER) #define modify_domain(dom,type) \ do { \ - unsigned int domain = current->thread.domain; \ - domain &= ~domain_val(dom, DOMAIN_MANAGER); \ - domain |= domain_val(dom, type); \ - current->thread.domain = domain; \ - set_domain(current->thread.domain); \ + unsigned long dacr = current->thread.dacr; \ + dacr &= ~domain_val(dom, DOMAIN_MANAGER); \ + dacr |= domain_val(dom, type); \ + current->thread.dacr = dacr; \ + set_dacr(dacr); \ } while (0) -#endif + +#endif /* !__ASM_PROC_DOMAIN_H */ diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/pgtable.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/pgtable.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/pgtable.h 2004-02-17 23:53:52.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/pgtable.h 2004-03-29 09:02:55.000000000 +1000 @@ -11,6 +11,7 @@ * now possible to combine ARM6, ARM7 and StrongARM versions. * 17-Apr-1999 RMK Now pass an area size to clean_cache_area and * flush_icache_area. + * 03-Oct-2003 JNZ Modified for FASS, based on Adam Wiggin's patches */ #ifndef __ASM_PROC_PGTABLE_H #define __ASM_PROC_PGTABLE_H @@ -101,6 +102,7 @@ #ifndef __ASSEMBLY__ +#include #include #include @@ -108,7 +110,7 @@ #define _PAGE_KERNEL_TABLE (PMD_TYPE_TABLE | PMD_DOMAIN(DOMAIN_KERNEL)) #define pmd_bad(pmd) (pmd_val(pmd) & 2) -#define set_pmd(pmdp,pmd) cpu_set_pmd(pmdp, pmd) +#define set_pmd(pmdp,pmd) cpd_set_pmd(pmdp, pmd) static inline pmd_t __mk_pmd(pte_t *ptep, unsigned long prot) { diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/pid.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/pid.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/pid.h 1970-01-01 10:00:00.000000000 +1000 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/pid.h 2004-03-29 09:02:55.000000000 +1000 @@ -0,0 +1,153 @@ +/******************************************************************************* +* Filename: include/asm-arm/proc-armv/pid.h * +* Description: ARM Porcess ID (PID) includes for Fast Address Space Switching * +* (FASS) in ARM Linux. * +* Created: 14/10/2001 * +* Changes: 19/02/2002 - Macros added. * +* Copyright: (C) 2001, 2002 Adam Wiggins * +******************************************************************************** +* This program is free software; you can redistribute it and/or modify * +* it under the terms of teh GNU General Public License version 2 as * +* published by the Free Software Foundation. * +*******************************************************************************/ + +#ifndef __ASM_PROC_PID_H +#define __ASM_PROC_PID_H + +#include + +/********** +* Defines * +**********/ + +#define ARMPID_SHIFT 25 + +/* First PID to be allocated first */ +#define ARMPID_START 1 + +/* Only 64 relocation spots on the StrongARM, see notes in arch/arm/mm/pid.c */ +#define ARMPID_MAX (1U << 6) + +/* Size of PID relocation area */ +#define ARMPID_TASK_SIZE (1UL << ARMPID_SHIFT) + +/* Start of dummy vm_area to seperate bss and stack */ +#define ARMPID_BRK_LIMIT (ARMPID_TASK_SIZE - MEGABYTE(1)) /* 31MB */ + +/* Number of CPD entries in the ARM PID relocation area */ +#define ARMPID_PTRS_PER_PGD (ARMPID_TASK_SIZE / PGDIR_SIZE) + +/* Mask to get rid of PID from relocated address */ +#define ARMPID_MASK (ARMPID_TASK_SIZE - 1) + +/********* +* Macros * +*********/ + +#define PIDNUM_TO_PID(pid_num) (pid_num << ARMPID_SHIFT) + +#define PID_TO_PIDNUM(pid) (pid >> ARMPID_SHIFT) + +/* Gets the ARM PID register value from a Modified Virtual Address (MVA) */ +#define MVA_TO_PID(mva) ((mva) & ~ARMPID_MASK) + +#define MVA_TO_PIDNUM(mva) (PID_TO_PIDNUM(MVA_TO_PID(mva))) + +#define MVA_TO_VA(mva) (mva & ARMPID_MASK) + +#define VA_TO_MVA(va) (va | get_armpid()) + +/* Find out the CPD index offset due to ARM PID relocation */ +#define ARMPID_CPD_OFFSET(pid) ((pid) >> PGDIR_SHIFT) + +/*********** +* Typedefs * +***********/ + +/* PID context */ +struct pid_struct { + unsigned int number; /* ARM PID number */ + unsigned int mm_count; /* Number of address-spaces using this ARM PID */ +}; + +/********** +* Externs * +**********/ + +extern struct pid_struct pids[ARMPID_MAX]; + +/******************* +* Inline Functions * +*******************/ + +/* Sets the CPU's PID Register */ +static inline void +set_armpid(unsigned int pid) +{ + __asm__ __volatile__( + "mcr p15, 0, %0, c13, c0, 0 @ set cpu dacr" + : : "r" (pid)); + +} /* set_armpid() */ + +/* Returns the state of the CPU's PID Register */ +static inline int +get_armpid(void) +{ + int pid; + + __asm__ __volatile__( + "mrc p15, 0, %0, c13, c0, 0 @ Get ARM PID" + : "=&r" (pid) : ); + + return (pid & (~ARMPID_MASK)); + +} /* get_armpid() */ + +static inline unsigned long +mva_to_va(unsigned long mva, struct mm_struct* mm_p) +{ + //printk("** mva_to_va: mm_p 0x%x mm_p->ctx.pid 0x%x CPU PID 0x%x **\n", + // mm_p, mm_p->context.pid, get_armpid()); + //printk("** mva_to_va: pid_num %d mva 0x%x **\n", + // PID_TO_PIDNUM(mm_p->context.pid), mva); + + if(get_armpid() && (get_armpid() == MVA_TO_PID(mva))) { + //printk("** mva_to_va: represents va 0x%x **\n", MVA_TO_VA(mva)); + + return MVA_TO_VA(mva); + } + + return mva; + +} /* mva_to_va() */ + +static inline unsigned long +va_to_mva(unsigned long va, struct mm_struct* mm_p) +{ + //printk("** va_to_mva: mm_p 0x%x mm_p->ctx.pid 0x%x CPU PID 0x%x **\n", + // mm_p, mm_p->context.pid, get_armpid()); + //printk("** va_to_mva: pid_num %d va 0x%x **\n", + // PID_TO_PIDNUM(mm_p->context.pid), va); + + if(va < ARMPID_TASK_SIZE) { + //printk("** va_to_mva: relocated to mva 0x%x **\n", VA_TO_MVA(va)); + + return VA_TO_MVA(va); + } + + return va; + +} /* va_to_mva() */ + +/*********************** +* Founction Prototypes * +***********************/ + +/* PID Manipulation */ + +void arm_pid_allocate(struct task_struct* task_p, struct mm_struct* mm_p); +void arm_pid_unallocate(struct pid_struct* pid_p); + + +#endif /* !__ASM_PROC_PID_H */ diff -ruN linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/processor.h linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/processor.h --- linux-2.4.24-uc0-snap1/include/asm-arm/proc-armv/processor.h 2004-02-17 23:53:52.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/asm-arm/proc-armv/processor.h 2004-03-29 09:02:55.000000000 +1000 @@ -14,6 +14,7 @@ * 09-09-1998 PJB Delete redundant `wp_works_ok' * 30-05-1999 PJB Save sl across context switches * 31-07-1999 RMK Added 'domain' stuff + * 09-10-2001 AGW Renamed thread.domain to thread.dacr to avoid confusion */ #ifndef __ASM_PROC_PROCESSOR_H #define __ASM_PROC_PROCESSOR_H @@ -45,10 +46,10 @@ #endif #define EXTRA_THREAD_STRUCT \ - unsigned int domain; + unsigned long dacr; #define EXTRA_THREAD_STRUCT_INIT \ - domain: domain_val(DOMAIN_USER, DOMAIN_CLIENT) | \ + dacr: domain_val(DOMAIN_USER, DOMAIN_CLIENT) | \ domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) | \ domain_val(DOMAIN_IO, DOMAIN_CLIENT) diff -ruN linux-2.4.24-uc0-snap1/include/linux/mm.h linux-2.4.24-uc0-snap1-fass0/include/linux/mm.h --- linux-2.4.24-uc0-snap1/include/linux/mm.h 2004-02-17 23:54:07.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/include/linux/mm.h 2004-03-29 09:02:55.000000000 +1000 @@ -78,6 +78,7 @@ struct file * vm_file; /* File we map to (can be NULL). */ unsigned long vm_raend; /* XXX: put full readahead info here. */ void * vm_private_data; /* was vm_pte (shared mem) */ + void * vm_sharing_data; /* Used by ARM FASS and IA-64 stuff */ }; #else /* NO_MM */ diff -ruN linux-2.4.24-uc0-snap1/kernel/fork.c linux-2.4.24-uc0-snap1-fass0/kernel/fork.c --- linux-2.4.24-uc0-snap1/kernel/fork.c 2004-02-17 23:54:12.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/kernel/fork.c 2004-03-29 09:02:55.000000000 +1000 @@ -184,6 +184,7 @@ tmp->vm_flags &= ~VM_LOCKED; tmp->vm_mm = mm; tmp->vm_next = NULL; + tmp->vm_sharing_data = NULL; file = tmp->vm_file; if (file) { struct inode *inode = file->f_dentry->d_inode; @@ -347,6 +348,10 @@ retval = -ENOMEM; mm = allocate_mm(); + + //printk("** copy_mm: task 0x%x, new mm_p 0x%x **\n", + // tsk, mm); /* awiggins */ + if (!mm) goto fail_nomem; @@ -770,6 +775,9 @@ struct task_struct *p; struct completion vfork; + //printk("** fork: current 0x%x mm_p 0x%x **\n", + // current, current->mm); /* awiggins */ + if ((clone_flags & (CLONE_NEWNS|CLONE_FS)) == (CLONE_NEWNS|CLONE_FS)) return -EINVAL; diff -ruN linux-2.4.24-uc0-snap1/mm/mmap.c linux-2.4.24-uc0-snap1-fass0/mm/mmap.c --- linux-2.4.24-uc0-snap1/mm/mmap.c 2004-02-17 23:54:12.000000000 +1100 +++ linux-2.4.24-uc0-snap1-fass0/mm/mmap.c 2004-03-29 09:02:55.000000000 +1000 @@ -1,6 +1,8 @@ /* * linux/mm/mmap.c * + * Modifications for TLB sharing in Linux (ARM/IA-64) (c) 2001 Adam Wiggins + * * Written by obz. */ #include @@ -106,8 +108,23 @@ vma->vm_next_share->vm_pprev_share = vma->vm_pprev_share; *vma->vm_pprev_share = vma->vm_next_share; } + + /* Deal with vm_sharing_data */ + arch_remove_shared_vm_struct(vma); } +#ifndef HAVE_ARCH_VM_SHARING_DATA + +/* Dummy function for default */ +static inline +arch_remove_shared_vm_struct(struct vm_area_struct* vma_p){} + +#else + +extern arch_remove_shared_vm_struct(struct vm_area_struct* vma_p); + +#endif + static inline void remove_shared_vm_struct(struct vm_area_struct *vma) { lock_vma_mappings(vma); @@ -522,6 +539,7 @@ vma->vm_pgoff = pgoff; vma->vm_file = NULL; vma->vm_private_data = NULL; + vma->vm_sharing_data = NULL; vma->vm_raend = 0; if (file) { @@ -836,6 +854,7 @@ mpnt->vm_pgoff = area->vm_pgoff + ((end - area->vm_start) >> PAGE_SHIFT); mpnt->vm_file = area->vm_file; mpnt->vm_private_data = area->vm_private_data; + mpnt->vm_sharing_data = NULL; if (mpnt->vm_file) get_file(mpnt->vm_file); if (mpnt->vm_ops && mpnt->vm_ops->open) @@ -1098,6 +1117,7 @@ vma->vm_pgoff = 0; vma->vm_file = NULL; vma->vm_private_data = NULL; + vma->vm_sharing_data = NULL; vma_link(mm, vma, prev, rb_link, rb_parent);