Increase swap space in OSX

[Paul Hillman]

I am using MatLAB with some large arrays and got an out of memory
error. 'help memory' stated to ask my system admin to increase my
swap space. Since the free space on my startup drive is limited, I
found a script that would move the swap space to a second harddrive
(I rebooted and verified that the swap files were now on the larger
drive.) My MatLAB program still runs out of memory at the same spot.
I ended up with about 6 swap files each about 76Megs in size. Any
help would be appreciated.

[Brett]

Have you calculated how much memory you are asking from Matlab? Is it
unreasonable?

Have you considered buying some more memory?

Brett.

Post by Paul Hillman
I am using MatLAB with some large arrays and got an out of memory
error. 'help memory' stated to ask my system admin to increase my
swap space. Since the free space on my startup drive is limited, I
found a script that would move the swap space to a second harddrive
(I rebooted and verified that the swap files were now on the larger
drive.) My MatLAB program still runs out of memory at the same spot.
I ended up with about 6 swap files each about 76Megs in size. Any
help would be appreciated.

[Paul Hillman]

Post by Brett
Have you calculated how much memory you are asking from Matlab? Is it
unreasonable?
Have you considered buying some more memory?

Yes, know the amount of memory requested is greater than physical
ram, shouldn't MatLAB or the OS swap things in and out using the swap
file as temporary storage? I thought that was the whole idea of
'virtual memory' to make the system think you had more ram than you
really did.

If I was going to need this extra amount of RAM a lot, yes I'd go out
and buy some, but I want to run a simulation with twice the number of
points in each dimension just as as a test case. I am not expecting
this to run fast, I expect it might take a week.

[Brett]

Post by Paul Hillman

Post by Brett
Have you calculated how much memory you are asking from Matlab? Is it
unreasonable?
Have you considered buying some more memory?

Yes, know the amount of memory requested is greater than physical
ram, shouldn't MatLAB or the OS swap things in and out using the swap
file as temporary storage? I thought that was the whole idea of
'virtual memory' to make the system think you had more ram than you
really did.

OK its greater than physical memory, but is it less than your allocated
virtual memory?

Have you tried to use the available memory more efficiently. ie. freeing
memory when it isn't required anymore?

Brett.

[Paul Hillman]

Post by Brett
OK its greater than physical memory, but is it less than your
allocated
virtual memory?
Have you tried to use the available memory more efficiently. ie. freeing
memory when it isn't required anymore?
Brett.

Yes, according to my calc, the requested array size is less than my
allocated virtual memory. (actually I am not trying this on both Mac
OSX and Windoze XP and not having much luck.

Actually the problem occurs in the beginning when I am just
allocating the arrays, for the large complex array I allocate them
with array=ones(n,n,3)*(1+i) for three nxn arrays. For n=1024, I
have not problems, but when I try n=2048 is when it seems to have
this memory problem. Note this array is roughly 201Megs.

[Paul Hillman]

My mistake, the declaration really is n=4096, so the array is
~804Megs.

Post by Paul Hillman
Actually the problem occurs in the beginning when I am just
allocating the arrays, for the large complex array I allocate them
with array=ones(n,n,3)*(1+i) for three nxn arrays. For n=1024, I
have not problems, but when I try n=2048 is when it seems to have
this memory problem. Note this array is roughly 201Megs.

[Richard Lang]

Post by Paul Hillman
My mistake, the declaration really is n=4096, so the array is
~804Megs.

Post by Paul Hillman
Actually the problem occurs in the beginning when I am just
allocating the arrays, for the large complex array I allocate them
with array=ones(n,n,3)*(1+i) for three nxn arrays. For n=1024, I
have not problems, but when I try n=2048 is when it seems to have
this memory problem. Note this array is roughly 201Megs.

Paul,

It could well be that the way you are doing the initialization is the
culprit. ones(n,n,3)*(1+i) will need to create an intermediate ones(n,n,3)
matrix in order to perform the multiplication and this means you are trying
to create 800 + 400 Megs. It is understandable that you might not have the
necessary contiguous memory addresses left after the first 400 is taken.

Post by Paul Hillman

Post by Paul Hillman
array = repmat(1+i, [n,n,3]);
array(n,n,3) = 1+i

The second is quicker but the array is mostly 0+0i. You'll have to fill the

Post by Paul Hillman

Post by Paul Hillman
array(:) = 1+i;

HTH
Richard

[Paul Hillman]

Post by Richard Lang
It could well be that the way you are doing the initialization is the
culprit. ones(n,n,3)*(1+i) will need to create an intermediate
ones(n,n,3)
matrix in order to perform the multiplication and this means you are trying
to create 800 + 400 Megs. It is understandable that you might not
have the
necessary contiguous memory addresses left after the first 400 is taken.

Post by Paul Hillman
array = repmat(1+i, [n,n,3]);
array(n,n,3) = 1+i

The second is quicker but the array is mostly 0+0i. You'll have to
fill the

Post by Paul Hillman
array(:) = 1+i;

HTH
Richard

Richard,
The finer points of matLAB I guess will take sometime to learn, and I
should have known that ones(n,n,3)*(1+i) would generate some
intermediate arrays.

Actually I am just trying to initialize the array to something
complex, so the second and faster way is the way to go.

I more or less solved the problem by generating each nxn array
separately and rewriting my program to access each (i.e. the third
index) a different way. My thought was what you alluded to, no matter
what size swap file you have, you must have contiguous memory to form
the array to begin with. So array(n,n,3) might run into difficulties
but array1(nxn), array2(nxn), array3(nxn) is OK.

[Tom Holt]

Paul,

I have no experience of OSX but have had similar problems under Linux
and Windows with Matlab. The solution is usually related to the
memory limitations of a 32-bit system coupled with the amount of
memory that modern operating systems grab for themselves. Increasing
the address space by upgrading to a 64-bit system should solve these
problems once and for all. I see that Apple has announced OSX Tiger
which purports to be a true 64-bit version of OSX (upgrade cost about
$120).

Without knowing full details of your system (amount of RAM, operating
system version, swap file details etc.) it is difficult know the best
solution to your problem. However, the long-term solution may be to
invest in a 64-bit operating system. This is what I have done for
Windows and Linux, and it works! Also, it is possible that the OSX
version of Matlab requires swapfile space to be contiguous. Have you
tried making a single swap file big enough for your data?

Tom

Post by Paul Hillman
I am using MatLAB with some large arrays and got an out of memory
error. 'help memory' stated to ask my system admin to increase my
swap space. Since the free space on my startup drive is limited, I
found a script that would move the swap space to a second harddrive
(I rebooted and verified that the swap files were now on the larger
drive.) My MatLAB program still runs out of memory at the same
spot.
I ended up with about 6 swap files each about 76Megs in size. Any
help would be appreciated.

[Paul Hillman]

Tom,
I think this is more of the problem, 2^32 is 4GB and the array size
I am trying to generate is about 800Megs, but with all the other
stuff the OS is keeping track of, this is probably it.

And I don't make the swap file, the OS does that. I just specify
where to put it.
Paul

Post by Tom Holt
Paul,
I have no experience of OSX but have had similar problems under Linux
and Windows with Matlab. The solution is usually related to the
memory limitations of a 32-bit system coupled with the amount of
memory that modern operating systems grab for themselves.
Increasing
the address space by upgrading to a 64-bit system should solve
these
problems once and for all. I see that Apple has announced OSX Tiger
which purports to be a true 64-bit version of OSX (upgrade cost about
$120).
Without knowing full details of your system (amount of RAM,
operating
system version, swap file details etc.) it is difficult know the best
solution to your problem. However, the long-term solution may be to
invest in a 64-bit operating system. This is what I have done for
Windows and Linux, and it works! Also, it is possible that the OSX
version of Matlab requires swapfile space to be contiguous. Have you
tried making a single swap file big enough for your data?
Tom

[Ken Prager]

Post by Paul Hillman
I am using MatLAB with some large arrays and got an out of memory
error. 'help memory' stated to ask my system admin to increase my
swap space. Since the free space on my startup drive is limited, I
found a script that would move the swap space to a second harddrive
(I rebooted and verified that the swap files were now on the larger
drive.) My MatLAB program still runs out of memory at the same spot.
I ended up with about 6 swap files each about 76Megs in size. Any
help would be appreciated.

Which version of Mac OS are you using? I'm guessing Jaguar (10.2.x)
because the way swap files are allocated was changed in Panther
(10.3.x). Under Panther, each new swap file is double the size of the
previous one (up to a certain size). This change was part of an overall
improvement to the memory management subsystem and may be enough to
solve your current problem.

Also, my experience is that there is a memory leak in Matlab and that it
doesn't return memory, even after quitting. I haven't tried to track
this down but know that as long as I don't run Matlab I stay at one swap
file. As soon as I start using Matlab I notice that my Mac adds a new
swap file every few days.

KP

[Paul Hillman]

Ken,
I am using Panther 10.3.8. All the swap files are generated when my
MatLAB M-file starts running.

Post by Paul Hillman

Post by Paul Hillman
I am using MatLAB with some large arrays and got an out of

memory

Post by Paul Hillman

Post by Paul Hillman
error. 'help memory' stated to ask my system admin to increase my
swap space. Since the free space on my startup drive is

limited,

Post by Paul Hillman
I

Post by Paul Hillman
found a script that would move the swap space to a second

harddrive

Post by Paul Hillman
(I rebooted and verified that the swap files were now on the

larger

Post by Paul Hillman
drive.) My MatLAB program still runs out of memory at the same

spot.

Post by Paul Hillman
I ended up with about 6 swap files each about 76Megs in size.

Any

Post by Paul Hillman

Post by Paul Hillman
help would be appreciated.

Which version of Mac OS are you using? I'm guessing Jaguar
(10.2.x)
because the way swap files are allocated was changed in Panther
(10.3.x). Under Panther, each new swap file is double the size of
the
previous one (up to a certain size). This change was part of an
overall
improvement to the memory management subsystem and may be enough to
solve your current problem.
Also, my experience is that there is a memory leak in Matlab and that it
doesn't return memory, even after quitting. I haven't tried to
track
this down but know that as long as I don't run Matlab I stay at one swap
file. As soon as I start using Matlab I notice that my Mac adds a
new
swap file every few days.
KP